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Abstract 


The  Educational  Quality  Indicators  (EQI)  initiative  was  a  three-year  collaboration 
between  Alberta  Education  and  twelve  school  jurisdictions  to  develop  indicator  systems 
to  measure  the  success  of  the  educational  enterprise.  Its  goal  was  to  assist  the  provincial 
government  and  school  jurisdictions  in  providing  better  information  to  improve  planning, 
policy  and  decision  making.  Towards  this  end,  ten  collaborative  action  research  projects 
were  initiated  in  1989  and  concluded  in  1992.  These  projects  developed  indicator 
systems  that  include  a  broad  range  of  outcomes,  methods  to  collect,  analyze  and  interpret 
data,  and  ways  to  report  and  communicate  the  information  to  different  audiences. 

Each  project  focused  on  specific  aspects  of  educational  improvement  and  accountability. 
The  projects  can  be  grouped  into  three  major  areas  -  system  accountability,  teaching  and 
learning,  and  alternative  student  assessment  strategies. 

Three  projects  focused  on  accountability.  The  Lacombe/Rocky  Mountain  project 
developed  a  comprehensive  school  system  review  process.  Grande  Prairie  developed 
school  and  district  profiles,  and  Lethbridge  Public  developed  a  collaborative  model  for 
school  and  program  evaluation. 

Two  projects  examined  teaching  and  learning.  Spirit  River  identified  26  behaviors 
of  effective  teaching  and  prepared  a  teacher  performance  baseline  for  the  division. 
Fort  McMurray  Catholic  implemented  the  principles  of  outcome-based  education 
in  mathematics. 

Five  projects  examined  alternative  ways  to  measure  student  success.  Edmonton  Public 
developed  mechanisms  for  portfolio  assessment  and  student  growth.  Calgary  Public 
and  Catholic  developed  indicators  of  quality  student  performance  in  art  and  mathematics. 
Brooks  explored  ways  to  measure  responsible  student  behavior.  Fort  McMurray  Public 
developed  a  three-level  approach  to  measuring  desirable  student  social  outcomes. 
Lethbridge  Catholic  developed  a  taxonomy  of  student  affective  behaviors. 

Achieving  Quality  is  the  final  report  of  the  Educational  Quality  Indicators  initiative. 
It  includes  a  summary  of  the  initiative  and  each  of  the  ten  projects.  A  final  chapter 
summarizes  the  major  accomplishments,  proposes  potential  provincial  indicators, 
and  suggests  further  steps  in  expanding  the  use  of  indicators  in  school  jurisdictions 
in  Alberta. 
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The  Educational  Quality  Indicators  Initiative 


Alberta  Education 

The  Educational  Quality  Indicators  (EQI)  initiative  was  a  collaboration  between 
Alberta  Education  and  twelve  school  jurisdictions  to  develop  and  implement 
indicator  systems  to  measure  the  success  of  the  educational  enterprise.  Its  goal 
was  to  assist  the  provincial  government  and  school  jurisdictions  in  providing 
better  information  to  improve  planning,  policy  and  decision  making.  Towards 
this  end,  ten  concurrent  collaborative  action  research  projects  were  initiated  in 
1989  and  concluded  in  1992.  These  projects  developed  indicator  systems  that 
include  a  broad  range  of  outcomes,  methods  to  collect,  analyze  and  interpret 
data,  and  ways  to  report  and  communicate  the  information  to  different  audiences. 
Achieving  Quality  is  the  final  report  of  the  EQI  initiative. 


Introduction 

Background 

A  new  wave  of  educational  reform  began  in  the  1980s.  Perceived  shortcomings 
in  education,  and  international  competition  in  all  sectors,  served  as  a  catalyst  for 
the  current  accountability  and  reform  movements  in  Canada  and  the  United  States, 
and  indeed  around  the  world.  These  reforms  resulted  in  many  organizations 
developing  or  implementing  what  have  become  known  as  educational  indicator 
systems.  Indicators  can  describe  education  in  a  way  that  permits  meaningful 
interpretation  to  take  place,  leading  to  educational  improvement  and  accountability. 

A  major  educational  reform  effort  in  Alberta  was  stimulated  by  a  series  of 
changes  in  the  political,  economic  and  social  climate  of  the  province.  In  1982, 
Alberta  Education  introduced  the  Achievement  Testing  Program  at  grades  3,  6 
and  9,  and  in  1984,  reinstated  the  Diploma  Examination  Program  at  grade  12  as 
part  of  the  requirements  for  high  school  graduation.  Annual  provincial  reports 
provide  information  about  test  results  at  the  provincial  level. 

In  1984,  the  Government  of  Alberta  introduced  a  number  of  other  initiatives 
to  improve  education  in  the  province.  These  included  the  School  Act  Review, 
the  Management  and  Finance  Plan,  the  Review  of  Secondary  School  Programs, 
and  the  introduction  of  five  evaluation  policies  (student,  teacher,  program,  school 
and  school  system).  As  a  result,  the  present  management  approach  attempts  to 
balance  the  inputs,  processes  and  outcomes  of  the  educational  enterprise. 
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Two  primary  beliefs  influenced  the  reform  initiatives  in  Alberta.  First,  the  student 
is  the  central  reason  for  schooling,  and  second,  education  is  a  purposeful  endeavor 
that  can  be  assessed.  To  assist  in  ensuring  that  these  beliefs  became  operational, 
Alberta  Education  promoted  a  results-based  approach  to  education  to  provide  a 
better  balance  among  inputs,  processes  and  results. 

The  EQI  initiative  was  one  of  the  projects  designed  to  promote  results-based 
education  in  Alberta.  A  significant  feature  was  that  it  introduced  a  collaborative 
model  to  the  provincial  reform  initiatives.  EQI  participants  became  members  of 
a  provincial  committee  which  discussed  and  recommended  directions  to  Alberta 
Education  in  achieving  results-based  education.  The  province  will  synthesize  the 
results  and,  together  with  its  other  initiatives,  work  toward  a  provincial  indicator 
system. 

EQI  was  designed  to  produce  a  set  of  indicators,  appropriate  standards  and 
accompanying  methods  to  measure  the  performance  of  the  educational  system 
in  the  province.  Indicators  provide  information  to  assist  in  assessing  the  quality  of 
educational  programs  and  the  delivery  system  by  focusing  on  student  outcomes. 
The  proposed  system  of  measuring  success  has  taken  into  consideration  and 
reflects  government  policy  and  the  goals  of  schooling.  It  addresses  two  essential 
questions: 

1.  Are  students  learning  to  their  potential? 

2.  Is  the  educational  system  supporting  student  learning  effectively  and 
efficiently? 


Rationale 

The  current  educational  reform  movement  led  to  expectations  for  greater 
accountability  and  increased  monitoring  and  evaluation  of  schools  and  systems. 
Indicator  systems  help  to  assess  the  effectiveness  and  efficiency  of  the  educational 
enterprise,  to  improve  education,  and  to  provide  a  mechanism  for  accountability. 
The  expected  results  over  the  three  years  of  the  EQI  initiative  were: 

•  indicator  systems  to  include  a  broad  range  of  student  outcomes,  points  of 
reference  for  comparing  results,  and  an  interpretative  framework  to  describe 
variation  among  students  and  schools; 

•  methods  to  collect,  analyze  and  interpret  the  information  provided  by  the 
indicators;  and 

•  ways  to  report  and  communicate  the  information  to  different  audiences. 

A  fundamental  principle  of  the  EQI  initiative  was  that  no  single  indicator,  or 
even  group  of  indicators,  could  fully  describe  the  complexity  of  education. 
Any  system  would  include  many  indicators,  measured  by  both  quantitative 
and  qualitative  methods,  for  selected  dimensions.  An  indicator  system  is 
intended  to  enhance  information  about  education  for  improved  action  in 
planning,  policy  and  decision  making. 
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The  use  of  multiple  indicators  repeated  over  time  can  describe  the  complexity 
of  education  and  provide  a  good  picture  of  performance.  Information  available 
through  documentation,  student  assessment,  surveys  and  observation,  can 
describe  the  effectiveness  and  efficiency  of  the  operation.  As  there  are  multiple 
users  of  information,  it  is  also  necessary  to  determine  what  type  of  information  the 
different  audiences  require  and  to  tailor  information  reports  to  their  specific  needs 
(Bock  &  Mislevy,  1988). 


Conceptual  Framework 

The  conceptual  framework  for  the  Educational  Quality  Indicators  initiative  guided 
the  direction  of  the  ten  collaborative  action  research  projects  and  the  discussion  of 
results-based  education  within  Alberta  Education.  Educational  indicators  provide 
information  about  a  system's  current  functioning,  suggest  whether  progress  is 
being  made  and  warn  of  any  potential  problems  (Oakes,  1986).  While  indicators 
cannot  describe  a  system  completely,  they  do  provide  a  picture  of  existing 
conditions  which  can  inform  planning,  policy  and  decision  making. 

An  educational  indicator  system  should  provide: 

•  information  that  is  feasible  to  gather,  valid  and  useful  for  policy  decisions; 

•  logical  clusters  of  indicators  to  increase  the  system's  interpretative  power; 

•  points  of  reference  such  as  a  previous  value,  the  value  of  a  comparison  group 
or  some  socially  determined  standard; 

•  measures  of  the  ubiquitous  features  of  schooling  that  are:  enduring,  easily 
understood,  feasibly  measured  and  generally  accepted  as  valid  and  reliable 
statistics; 

•  a  reporting  system  that  is  accurate  and  timely  (Oakes,  1986;  Selden,  1988). 

A  four-dimensional  model  of  education  was  developed  to  guide  the  direction  of 
this  initiative.  It  consists  of  partners  (schooling,  family  and  society),  conditions 
(context,  inputs  and  processes),  student  outcomes  (cognitive,  affective  and 
behavioral)  and  time  (grades  3,  6, 9  and  12)  (McEwen  &  Zatko,  1989).  The 
model  draws  on  the  work  of  Carroll  (1963),  Hymel  (1988),  Walberg  (1984), 
Oakes  (1986)  and  Shavelson  et  al  (1987).  The  EQI  model  extends  the  work  of 
the  above  by  consolidating  the  joint  responsibility  of  the  partners  who  contribute 
specific  conditions  to  developing  student  learning  over  time. 

The  model  provides  a  way  of  describing  the  interrelationships  among  the 
dimensions.  This  description  of  the  educational  system  and  student  achievement 
is  not  meant  to  predict  future  behavior,  but  rather  to  help  in  providing  some 
understanding  of  the  educational  phenomena  that  can  subsequently  be  acted 
upon  to  improve  the  situation.  It  is  intended  to  permit  multiple  levels  of  analysis 
-  province,  community,  school,  classroom  and  individual  students  -  so  that  a 
range  of  interpretations  can  be  drawn  and  improvement  targets  proposed  (Cziko, 
1989). 
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Education  is  a  complex  social  enterprise.  Focusing  on  schooling  alone,  without 
regard  for  the  other  influences  that  shape  its  direction,  will  likely  not  result  in 
significant  change.  This  model  incorporates  the  responsibilities  of  the  other  major 
partners  and  recognizes  conditions  which  contribute  to  student  outcomes.  For 
educational  constituencies  to  determine  whether  or  not  students  are  making 
appropriate  progress,  it  is  necessary  to  examine  results  over  time  to  determine 
what  changes  should  and  have  been  effected. 

The  EQI  initiative  set  three  criteria  for  a  successful  indicator  system:  an 
interpretative  framework  to  describe  variation  among  students  and  schools; 
student  outcomes  related  to  the  educational  enterprise;  and  points  of  reference 
for  comparing  results  (Alberta  Education,  1988). 

•  The  interpretative  framework  consists  of  the  partners  (schooling,  family  and 
society)  and  conditions  (context,  inputs  and  processes).  Together  the  partners 
and  their  respective  conditions  provide  a  framework  for  helping  to  understand 
the  student  outcomes  which  schools,  the  family  and  society  strive  to  produce 
through  adjusting  the  inputs  and  processes;  the  context  is  not  as  readily 
changeable.  School  jurisdictions  and  government  collect  a  lot  of  information 
on  context  and  inputs;  some  also  collect  information  on  instructional 
processes.  School  jurisdictions  are  interested  in  results  for  students  in  other 
jurisdictions  of  comparable  size,  location  and  economic  base. 

•  Student  outcomes  include  a  broad  range  of  the  behaviors  desired  by  schools  in 
the  cognitive,  affective,  physical  and  social  domains;  the  outcomes  selected 
for  measurement  should  relate  to  the  goals  and  priorities  of  the  local  school 
jurisdiction  and  the  province. 

•  Points  of  reference  include  time,  groups  and  targets.  Assessment  of  selected 
indicators  can  occur  either  on  an  annual  or  a  periodic  basis;  for  example, 
Alberta  achievement  tests  and  diploma  examinations  are  administered 
annually  to  students  in  grades  3,  6,  9  and  12,  whereas  assessments  such 

as  those  conducted  by  the  International  Association  for  the  Evaluation  of 
Educational  Achievement  (IEA)  and  the  International  Assessment  of 
Educational  Progress  (IAEP)  occur  periodically  in  selected  subject  areas. 
Groups  for  comparing  results  can  include  local,  provincial,  national  and 
international  counterparts.  Targets  include  benchmarks  and  standards. 
Benchmarks  describe  the  existing  level  of  conditions  and  outcomes  whereas 
standards  define  an  optimal  or  desired  level;  targets  specify  a  feasible  level 
of  improvement  within  a  predetermined  period  of  time.  A  target  sets  an 
improvement  increment  over  what  exists  (the  benchmark)  in  an  attempt  to 
reach  what  is  desired  (the  standard). 


Design 

The  Terms  of  Reference  (November  1988)  provided  direction  to  the  EQI  projects. 
School  jurisdictions  that  were  already  concerned  with  measuring  educational 
quality  were  identified  and  invited  to  submit  a  proposal.  Alberta  Education 
provided  the  funds  to  assist  these  jurisdictions  to  improve  their  assessment 
procedures  and  to  share  the  results  with  others  in  the  province. 
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The  1988-89  school  year  was  devoted  to  developing  the  conceptual  framework 
and  implementation  plan,  developing  the  Terms  of  Reference  for  the  projects, 
initiating  the  collaborative  action  research  projects,  and  organizing  the 
committees.  The  projects  began  in  1989  and  formally  concluded  in  1992. 

Coordination  of  the  EQI  initiative  resided  in  the  Policy  and  Planning  Branch  of 
Alberta  Education.  A  provincial  coordinator  was  responsible  for  the  initiative. 
Two  committees  also  provided  assistance  to  EQI.  The  Working  Group  consisted 
of  the  liaison  staff  from  each  of  the  five  Regional  Offices  of  Education  (located  in 
Grande  Prairie,  Edmonton,  Red  Deer,  Calgary  and  Lethbridge).  This  committee 
was  responsible  for  monitoring  the  projects.  It  met  on  a  regular  basis  to  discuss 
progress,  identify  issues  and  strategies,  and  to  plan  the  provincial  meetings.  The 
second  committee  was  the  interdivisional  Advisory  Group  with  representatives 
from  branches  within  the  Student  Programs  and  Evaluation  and  Finance  and 
Administration  Divisions  of  Alberta  Education;  it  met  on  a  periodic  basis  and 
provided  input  and  feedback.  Both  committees  participated  in  the  semiannual 
provincial  meetings. 


Local  Development 

Each  project  was  autonomous  and  the  responsibility  of  a  local  coordinator  and 
project  team.  This  coordinator  reported  to  the  superintendent  for  the  school 
jurisdiction  and  was  responsible  for  the  developmental  work  and  the  management 
of  the  tasks  identified  in  the  contract  between  the  school  jurisdiction  and  the 
department.  The  contract  required  annual  progress  reports  as  well  as  a  final  report 
upon  completion  of  the  project  in  1992.  Each  project  was  guided  by  a  local 
Steering  Committee  consisting  of  the  local  coordinator  and  other  staff  from  the 
participating  school  jurisdiction,  liaison  staff  from  the  Regional  Office  of 
Education,  and  the  provincial  coordinator.  The  Steering  Committee's 
responsibilities  included  consultation  and  evaluation  of  the  terms  of  the  contract. 

Each  participating  school  jurisdiction  was  responsible  for  its  own  project  because 
each  represents  a  community  which  has  values  it  wishes  to  impart  to  its  students 
through  its  goals,  priorities  and  expectations.  It  was  important  that  each  project 
reflect  these  aspirations  by  involving  its  public  -  students,  teachers,  administrators 
trustees,  parents,  others  -  in  discussions  on  which  indicators  to  include  and  why. 
Without  input  from  the  larger  community,  the  indicators  might  not  provide  a 
representative  picture  of  the  expected  outcomes  of  the  educational  system. 

Each  participating  school  jurisdiction  developed  and  implemented  a  local 
indicator  system  having  three  components:  (1)  a  set  of  indicators  including  an 
interpretative  framework,  student  outcomes  and  points  of  reference;  (2)  methods 
to  collect,  analyze  and  interpret  the  information;  and  (3)  a  reporting  and 
dissemination  strategy  to  inform  diverse  audiences  of  the  results.  The  selected 
indicators  represent  each  jurisdiction's  goals  and  priorities  and  reflect  the 
community's  expectations.  Every  project  had  three  phases,  each  of  approximately 
one  year's  duration.  The  first  year,  1989-90,  was  a  developmental  one  and  the 
subsequent  two  years  resulted  in  field-testing  the  prototype  sets  of  indicators  and 
procedures,  and  then  refining  them. 

School  jurisdictions,  if  they  are  to  make  sense  of  the  outcomes  of  their  educational 
enterprise,  need  to  interpret  the  results  in  terms  of  their  local  circumstances.  In 
the  process  of  developing  a  preliminary  set  of  indicators,  trying  it  out  in  their  own 
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jurisdiction  and  refining  it  on  the  basis  of  the  results,  jurisdictions  took  the 
opportunity  to  develop  and  use  indicators  that  are  flexible,  responsive  to  local 
needs,  have  community  support,  and  can  provide  them  with  the  type  of 
information  they  need  for  setting  policies,  practices  and  procedures. 


Provincial  Support 

Alberta  Education  supported  this  initiative  by  providing  resources  (funds  and 
consultation),  information  and  coordination.  Alberta  Education  committed  $1 
million  to  the  EQI  initiative  to  support  the  three-year  concurrent  research  projects. 

The  Collaborator  -  This  quarterly  newsletter  was  introduced  as  a  communications 
mechanism  for  the  project  teams  and  Alberta  Education.  The  first  issue  was 
released  in  September  1989  at  the  first  provincial  meeting.  It  provided  an 
overview  of  EQI  and  each  of  the  ten  concurrent  collaborative  action  research 
projects.  Eleven  subsequent  issues  were  published.  Each  June  issue  contained 
an  annual  status  report.  EQI  participants  regularly  contributed  material  on  the 
progress  of  their  projects.  This  92-page  record  documented  events  and  activities 
over  the  three  years. 

Technical  Documents  -  Alberta  Education  developed  three  technical  documents 
to  support  the  EQI  initiative.  The  first,  an  Annotated  Bibliography »,  was  initially 
distributed  in  July  1989.  It  was  subsequently  revised,  expanded  and  reorganized 
and  a  second  edition  released  in  January  1990,  followed  by  a  supplement  in 
December  1990.  This  second  edition  contains  approximately  350  annotations 
organized  into  14  themes  under  three  major  headings:  indicator  systems, 
interpretative  framework  and  outcomes.  It  has  been  distributed  to  all  school 
jurisdictions  in  Alberta  to  promote  the  discussion  of  results-based  education. 
The  second  report,  Methodological  Considerations,  consists  of  a  rationale  for 
a  balanced  approach  to  quantitative  and  qualitative  analysis,  followed  by 
considerations  in  employing  these  two  methods.  The  third  document,  Inventory 
of  Assessment  Instruments,  contains  a  description  of  technical  criteria  for 
analyzing  instruments,  and  critiques  of  more  than  90  instruments.  The  inventory 
identified  potential  assessment  instruments  and  screened  them  according  to  their 
psychometric  properties  and  suitability  for  EQI.  It  provided  participants  with  a 
convenient  pool  of  possible  measures  for  the  types  of  indicators  that  interested 
them. 

Semiannual  Meetings  -  The  key  participants  -  Alberta  Education  staff,  project 
coordinators  and  superintendents  -  met  semiannually  to  share  information,  discuss 
and  interpret  findings,  and  identify  issues  and  strategies.  These  meetings  served 
to  maintain  personal  contact  among  participants,  to  promote  synergy,  and 
encouraged  people  to  meet  deadlines. 

Professional  Development  -  An  important  component  of  the  EQI  initiative  was 
professional  development.  The  November  1990  meeting  featured  concurrent 
workshops  on  the  Inventory  of  Assessment  Instruments  and  qualitative  methods 
described  in  Methodological  Considerations.  These  sessions  provided  participants 
with  the  opportunity  to  work  through  the  material,  ask  questions,  and  discuss 
issues.  The  interactive  computer  version  of  the  Annotated  Bibliography  was 
also  demonstrated.  The  January  1992  meeting  featured  clarifying  final  report 
expectations  and  criteria  for  ensuring  quality  reports.  There  were  also  two 
concurrent  workshops.  The  first  featured  data  analysis  and  presentation  using  a 
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statistical  and  graphic  utility;  participants  were  able  to  try  out  Stat  View  on  five 
Macintosh  microcomputers  and  a  PowerBook.  The  second  workshop  focused 
on  communicating  with  the  public;  it  involved  developing  a  communications 
strategy  and  group  work  in  developing  strategies  for  the  projects. 

Technical  Advice  -  In  1991,  two  measurement  experts  were  engaged  to  provide 
technical  advice  to  the  project  teams.  University  of  Alberta  professors  Tom 
Maguire  and  Todd  Rogers  reviewed  the  materials  prepared  by  the  teams  and, 
together  with  the  provincial  coordinator,  visited  each  team  to  discuss  the  projects. 
Each  team  received  written  comments  on  its  project  and  suggestions  for 
improvement.  This  individualized  assistance  supplemented  that  provided 
by  the  provincial  coordinator. 


The  Projects 

Achieving  Quality  presents  a  summary  of  each  of  the  ten  projects  that  have 
devised  many  new  indicators  and  practices  for  assessing  the  quality  of  education 
in  their  districts.  The  new  knowledge  generated  by  the  projects  can  be  used  by 
others.  The  summaries  have  been  organized  into  three  major  areas  -  system 
accountability,  teaching  and  learning,  and  alternative  student  assessment 
strategies,  each  corresponding  to  the  major  focus  of  the  projects.  Within  each 
area,  there  is  diversity  in  types  of  indicators,  approaches  and  reporting  results. 


System  Accountability 

Three  projects  focused  on  accountability  at  the  district  level.  They  developed 
indicator  systems  to  examine  the  district  as  a  whole,  at  the  school  level,  and 
for  specific  programs. 

The  Lacombe/Rocky  Mountain  Project  developed  a  comprehensive  school  system 
review  process  encompassing  student,  teacher,  program,  school  and  school  system 
evaluations.  Its  design  consisted  of  five  interrelated  components:  involvement  of 
educational  partners  in  developing  and  implementing  the  review,  development  of  a 
set  of  goals  of  learning  to  form  the  basis  of  the  review,  identification  of  outcome, 
condition  and  process  indicators,  identification  of  ways  of  measuring  the  indicators 
and  interpreting  the  results,  and  a  planning  process  to  use  the  findings. 

Grande  Prairie  Public  developed  school  and  district  profiles  to  inform  stakeholders 
of  the  quality  of  education  in  its  district.  The  profiles  provide  information  in  four 
areas  -  student  achievement,  school  climate,  quality  of  instruction,  and  funding  - 
and  will  be  published  annually. 

Lethbridge  Public  developed  a  collaborative  model  for  school  and  program 
evaluation.  The  model  includes  inputs,  student  outcomes,  and  identifies 
assessment  processes.  It  is  generalizable  and  could  be  adapted  for  other 
programs. 
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Teaching  and  Learning 

Two  projects  focused  on  teaching  and  learning  -  one  examined  teaching  behaviors 
and  another  outcome-based  education. 

Spirit  River  identified  26  behaviors  of  effective  teaching  and  grouped  them  into 
four  areas  -  planning  and  preparation,  instructional  strategies,  communication 
skills,  and  pupil/teacher  relationships.  In  1991,  teachers  were  observed  on  these 
behaviors  and  a  composite  performance  baseline  prepared.  Student  perceptions, 
attitudes,  achievement  and  discipline  data  were  also  collected.  Data  collection 
was  repeated  in  1992  to  provide  a  basis  for  comparison. 

Fort  McMurray  Catholic  implemented  the  principles  of  outcome-based  education 
in  mathematics  from  grades  3  to  10,  and  measured  student  success  in  achievement, 
responsibility  and  attitudes. 


Alternative  Student  Assessment  Strategies 

Five  projects  examined  alternative  ways  to  measure  student  success.  These 
included  portfolio  assessment  and  growth,  exemplars  of  performance,  responsible 
student  behavior,  social  skill  development,  and  affective  outcomes. 

Edmonton  Public  developed  mechanisms  for  student  portfolio  assessment  and 
computer  technology  for  communicating  and  storing  information  about  student 
growth,  and  established  and  validated  curriculum-based  district  level  assessment 
strategies. 

Calgary  Public  and  Catholic  developed  indicators  of  quality  student  performance 
in  art  and  mathematics.  EQI-Art  focused  on  the  product  of  learning,  that  is,  the 
artwork  prepared  by  the  student,  while  EQI-Math  focused  on  the  process  of 
student  thinking  in  solving  mathematical  problems. 

Brooks  explored  ways  to  measure  responsible  student  behavior.  This  project 
collected  data  from  a  variety  of  sources  and  involved  the  community  in 
reaffirming  its  commitment  to  positive  student  behavior. 

Fort  McMurray  Public  focused  on  desirable  student  social  outcomes.  The  three- 
level  approach  to  measuring  social  skills  includes  a  diagnostic  screen  and  a 
manual  of  activities  to  help  students  develop  effective  personal,  interpersonal, 
social  reasoning  and  classroom  behaviors. 

Lethbridge  Catholic  developed  a  taxonomy  of  student  affective  behaviors  in  five 
areas  -  self-worth,  interpersonal  relationships,  world  awareness,  learning,  and 
spiritual  life. 
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Overview 


This  first  chapter  has  introduced  the  Educational  Quality  Indicators  initiative.  It  has 
described  the  background  and  rationale  for  this  partnership  with  participating  school 
jurisdictions.  The  conceptual  framework  guided  the  direction  of  the  ten  projects  and 
provided  a  common  framework  and  language  for  results-based  education.  Alberta 
Education  supported  the  projects  with  resources  (funds  and  consultation),  information 
and  coordination. 

Each  project  selected  aspects  of  the  indicator  system  based  on  local  goals,  priorities 
and  expectations.  The  ten  projects  have  been  organized  into  three  major  areas 
dealing  with  system  accountability,  teaching  and  learning,  and  alternative  student 
assessment  strategies.  The  next  ten  chapters  of  Achieving  Quality  provide  a  summary 
of  each  project.  A  final  chapter  discusses  EQI  accomplishments  and  next  steps. 
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School  System  Review 


A  Comprehensive  Process 

County  ofLacombe  No.  14  and 
Rocky  Mountain  School  Division  No.  15 

The  County  ofLacombe  and  Rocky  Mountain  School  Division  initiated  a  study  to 
develop  and  implement  a  comprehensive  school  system  review  based  on  quality 
indicators  of  student  learning  and  indicators  of  related  conditions  and  processes. 
A  further  purpose  was  to  combine  teacher,  program,  school  and  school  system 
evaluations  within  the  review  process.  The  design  of  the  project  consisted  of  five 
interrelated  components:  involvement  of  educational  partners  in  developing  and 
implementing  the  review,  development  of  a  set  of  goals  of  learning  to  form  the 
basis  of  the  review,  identification  of  outcome,  condition  and  process  indicators, 
identification  of  ways  of  measuring  the  indicators  and  interpreting  the  results, 
and  a  planning  process  to  utilize  the  findings. 

The  components  of  the  review  were  collaboratively  developed  over  a  two-year 
period.  A  balanced  set  of  goals  in  four  groups  -  intellectual,  social,  personal 
and  vocational  learning  -  was  developed  and  indicators  of  cognitive,  affective  and 
behavioral  learning  were  identified.  Nine  significant  measures  were  identified  or 
developed  for  the  outcome  indicators  whereas  four  measures  and  six  processes 
were  selected  for  condition  and  process  indicators.  Locally  developed  instruments 
and  other  processes  were  field-tested. 

The  full  system  review  was  implemented  in  the  third  year.  Data  were  collected 
and  have  been  assembled,  and  a  procedures  handbook  on  implementing  the  review 
has  been  completed.  Individual  teacher  reports  on  classroom  conditions  and 
processes  and  school  and  systems  reports,  organized  by  the  goals  of  learning, 
have  also  been  finalized. 

The  review  process  was  found  to  be  sound  and  feasible  to  implement  The  findings 
on  student  learning  reveal  that  a  comprehensive  description  of  student  learning  by  a 
system  of  indicators  is  possible. 

To  date  the  interpretation  and  planning  processes  have  not  been  completed,  but  both 
boards  are  committed  to  using  the  results  in  a  planning  process  followed  by  four 
years  of  implementing  improvements. 

The  review  provides  better  data  for  accountability  and  more  definitive  data  for 
planning.  It  balances  evaluation,  planning  and  implementation  activities  more 
appropriately  and  uses  data  for  a  variety  of  reports.  The  process  could  be  adapted 
for  use  in  other  school  systems  to  yield  similar  benefits. 
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Introduction 


The  County  of  Lacombe  No.  14  and  the  Rocky  Mountain  School  Division  No.  15, 
two  small  rural  school  jurisdictions  in  central  Alberta,  have  had  a  continuing  interest 
in  being  accountable  and  in  improving  the  quality  of  education.  Commencing  in 
1985,  both  systems  have  conducted  satisfaction  surveys  with  their  publics,  reported 
the  results  and  established  improvement  plans.  Additionally,  to  adhere  to 
departmental  policy,  the  County  of  Lacombe  developed  an  integrated  teacher  and 
school  evaluation  process  whereas  Rocky  Mountain  School  Division  concentrated 
on  more  traditional  teacher,  school  and  program  evaluations.  Both,  as  is  usual, 
reviewed  and  reported  on  student  achievement  tests  and  diploma  exams.  Hence 
when  the  provincial  initiative  on  educational  quality  indicators  became  a  reality,  they 
welcomed  the  opportunity  to  move  to  another  step  in  the  evolution  of  evaluation  in 
their  systems  and  to  develop  a  comprehensive  school  system  review  process  based 
on  educational  quality  indicators  that  would  combine  all  components  of  evaluation 
into  one  package. 


Rationale 

Although  both  school  systems  had  been  devoting  an  extensive  amount  of  time  to 
a  variety  of  commendable  evaluation  activities,  they  did  not  see  concomitant 
improvements  being  made  in  the  educational  program.  Part  of  the  reason  for  this 
lack  of  program  improvement  was  that  supervisory  personnel  and  school  staffs 
were  devoting  most  of  their  time  and  effort  to  evaluation  with  little  time  left  for 
developing  and  implementing  follow-up  improvements.  The  publics  involved  in 
the  surveys  and  other  evaluative  activities  were  also  concerned  about  a  lack  of 
benefits  from  these  activities.  System  personnel  were  also  cognizant  of  the  many 
times  when  educational  change  in  system  decisions  were  precipitated  by  results  on 
a  single  indicator  or  the  occurrence  of  a  single  incident. 

System  personnel,  nevertheless,  felt  that  many  worthwhile  evaluation  activities 
had  been  developed  in  the  systems:  the  public  reporting  of  student  results,  the 
satisfaction  surveys  and  the  teacher,  program  and  school  evaluations.  Thus,  the 
systems  wanted  to  address  the  concerns,  utilizing  existing  evaluation  activities  but 
to  evolve  further  by  basing  evaluations  on  educational  quality  indicators.  They 
wanted  to  develop  a  more  comprehensive  evaluation  system,  one  that  was  efficient 
and  that  included  a  process  for  planning  and  implementing  improvements. 


Purposes 

The  primary  purpose  of  the  study  was  to  develop  a  comprehensive  school  review 
process  based  on  student  outcome  indicators  and  the  condition  and  process 
indicators  that  contribute  to  student  learning.  Thus,  the  review  process  would  not 
only  determine  how  well  students  were  learning,  but  would  also  determine  the 
conditions  and  processes  that  were  contributing  to  learning  and  those  that  were 
not.  The  findings  would  be  utilized  to  develop  a  plan  for  enhancing  student  learning 
and  to  inform  the  public  about  the  state  of  learning  in  the  system. 

Another  purpose  was  to  develop  an  efficient  evaluation  system  that  would  maximize 
the  use  of  the  findings  to  generate  teacher,  school,  program  and  school  reports. 
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Design 

The  school  system  review  process  is  based  on  an  educational  quality  indicator 
system  that  focuses  on  indicators  of  student  outcomes  as  well  as  indicators  of 
conditions  and  processes  that  affect  student  outcomes.  A  review  of  the  literature 
suggests  that  an  indicator  system  should: 

•  be  based  on  a  comprehensive  statement  of  goals  of  learning  which  are  a  local 
interpretation  of  provincial  goals  (David,  1986); 

•  include  multiple  indicators  that  comprise  a  system  to  increase  interpretative 
power  (Goertz,  1989); 

•  include  a  point  of  comparison  such  as  a  larger  group  or  a  previous  result, 
and  include  conditions  and  processes  to  assist  in  interpreting  outcomes  (US 
Department  of  Education,  1985;  Goertz,  1989;  Haertel,  Katzenmeyer,  & 
Haertel,  1989); 

•  focus  on  indicators  that  are  enduring,  easily  understood  and  feasibly  measured 
(Oakes,  1989); 

•  produce  information  that  can  be  readily  used  for  planning  improvements  or 
making  policy  decisions  (US  Department  of  Education,  1985); 

•  involve  all  educational  partners  in  the  development  and  implementation  of  the 
indicator  system  (David,  1986); 

•  be  designed  as  an  integral  part  of  the  operation  of  the  school  system  (Kaufman, 
1988). 

The  design  of  the  school  system  review  process,  incorporating  the  above 
characteristics  of  an  indicator  system,  consists  of  five  distinct  but  interrelated 
components: 

1 .  a  plan  for  involvement  of  all  educational  partners  in  its  development  and 
implementation; 

2 .  a  comprehensive  statement  of  goals  of  learning; 

3 .  a  system  of  indicators  to  measure  all  goals  of  learning  and  the  contributing 
conditions  and  processes; 

4 .  a  process  for  measuring  and  interpreting  learning; 

5 .  a  planning  process. 
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These  components  are  shown  in  Figure  1  below  and  are  described  further  following 
the  diagram. 


EducationaC 

(Partner 
(Participation 


Goals  of  Student  Learning 


) 


Outcome,  Condition  and 
Process  Indicators 


Measurement  and 
Interpreting  Learning 


A  Plan  for  the  Future 


Figure  1 :  The  Project  Blueprint 


Educational  Partner  Involvement 

The  educational  partners  were  involved  in  the  development  and  implementation  of 
all  components  of  the  school  system  review  process:  the  specification  of  goals, 
identification  of  indicators  and  comparators,  the  development  of  some  of  the 
measures,  the  interpretation  of  the  results  and  the  development  of  an  improvement 
plan.  There  were  generally  four  types  of  involvement  by  the  educational  partners. 
First,  there  was  a  steering  committee,  representing  a  broad  cross  section  of  the 
educational  community,  in  each  jurisdiction  that  planned  and  sanctioned  directions 
for  the  project  Second,  a  process  committee  of  parents,  teachers  and 
administrators  in  each  system  was  involved  in  the  detailed  development  and 
implementation  of  the  review  system.  Third,  there  was  direct  and  wide 
involvement  of  parents,  staff  and  the  public  in  some  of  the  developments. 
Fourth,  administrators  and  staff  were  used  as  a  sounding  board  as  the  process 
was  being  developed. 
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Goals  of  Student  Learning 


The  development  of  a  comprehensive  statement  of  goals  of  student  learning  was 
required  to  provide  a  focus  for  the  school  system  review.  As  a  result,  the 
evaluation  could  then  be  based  on  the  specific  results  that  each  system  wished  to 
achieve  on  a  local  interpretation  of  provincial  goals.  To  accomplish  this  statement 
of  goals,  the  steering  committee,  through  a  group  process  and  using  assessment 
guidelines  provided  by  the  coordinator,  reviewed  a  number  of  goal  statements  from 
the  department  Although  consensus  was  achieved  at  this  stage  on  a  goal  statement 
in  each  jurisdiction,  the  statements  were  refined  when  the  participants  in  future 
meetings  were  identifying  outcome  and  process  indicators.  Goals  were  found  to 
overlap  and  some  goals  were  found  not  to  be  measurable.  At  this  stage,  it  was  also 
suggested  that  the  goals  be  put  into  four  groups:  intellectual,  social,  personal  and 
vocational.  It  was  felt  that  this  categorization  would  enhance  understanding  of 
the  goals  and  promote  the  appropriate  emphasis.  Furthermore,  the  goals  were 
categorized  on  the  basis  of  responsibility  for  implementing  them  -  primary  for 
school  responsibility  and  shared  for  both  school  and  family  responsibility. 

The  significance  of  this  exercise  of  developing  a  goal  statement  is  that  the 
jurisdictions  now  have  a  statement  of  goals  of  learning  with  a  local  flavor  that 
educational  partners  understand  and  accept  as  a  basis  for  educational  programs 
and  their  evaluation.  The  goals  that  were  agreed  upon  and  sanctioned  by  the  local 
steering  committees  are  outlined  in  Figure  2. 


Outcome,  Condition  and  Process  Indicators  and  Measures 

Utilizing  the  statement  of  goals  of  student  learning,  indicators  of  achievement  of  the 
goals,  appropriate  existing  or  required  measures  and  comparators  were  identified  at 
meetings  of  parents  and  teachers  at  all  schools  in  both  systems. 

The  meetings  also  succeeded  in  increasing  awareness  and  support  for  the  project  as 
well  as  the  use  of  multiple  indicators  since  there  was  a  general  concern  about  using 
single  indicators  to  describe  the  quality  of  education. 

Through  the  workshop  sessions  and  subsequent  revisions  by  project  personnel  and 
committees,  outcome  indicators  relevant  to  any  jurisdiction  review  were  identified. 
The  indicators  were  organized  into  three  categories:  cognitive,  affective  and 
behavioral.  The  cognitive  indicators  included  achievement,  program  participation, 
completion  rates,  independent  learning  and  problem  solving;  the  affective  indicators 
were  student  satisfaction,  self-concept,  attitudes,  motivation  and  aspirations;  the 
behavioral  indicators  were  desirable  characteristics,  participation  in  related 
activities,  community  service,  attendance,  image,  health  and  fitness. 

A  sample  of  the  outcome  indicators,  measures  and  comparators  for  the  goal  to 
"develop  knowledge,  skills  and  attitudes  in  language  arts  and  communication"  is 
contained  in  Figure  3.  A  similar  outline  of  indicators,  measures  and  comparators 
is  available  for  every  goal  of  learning. 
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Intellectual 
learning 


Develop  intellectual  curiosity  and  a  desire  for  independent  lifelong 
learning,  (shared) 

Develop  knowledge,  skills  and  positive  attitudes  in  language  and 
communication,  mathematics,  practical  and  fine  arts,  sciences  and 
social  studies,  (primary) 

Develop  ways  of  learning  including: 

•  skills  of  finding,  comprehending,  organizing,  analyzing,  applying  and 
synthesizing  information; 

•  skills  of  learning  through  technology; 

•  skills  of  studying,  (primary) 


Social  learning 


Develop  a  sense  of  community  responsibility  which  embraces: 

•  an  understanding  of  and  appropriate  participation  in  citizenship  at  the 
local,  national,  and  international  levels;  (primary) 

•  respect  for  law  and  authority,  public  and  private  property,  and  the 
opinions  and  rights  of  others;  (shared) 

•  appreciation  of  the  importance  of  tradition  and  culture;  (shared) 

•  the  careful  use  of  natural  resources  and  the  preservation  of  the 
physical  environment,  (shared) 


Personal  learning 


Acquire  knowledge  and  develop  skills  which  contribute  to  physical 
well-being,  (shared) 

Acquire  knowledge,  develop  skills,  attitudes,  and  habits  that  contribute 
to  emotional  well-being  including: 

•  achieving  a  positive  self-concept; 

•  acquiring  a  high  level  of  self-discipline  and  individual  responsibility; 

•  acquiring  an  ability  to  respond  to  change; 

•  developing  short  and  long-term  personal  goals,  (shared) 


Vocational 
learning 


Acquire  knowledge  and  skills,  attitudes,  and  habits  for  individuals  to  be 
successful  in  the  world  of  work,  (primary) 


Figure  2 :  Goals  of  Student  Learning 
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Intellectual  Learning:  Develop  knowledge,  skills  and  positive  attitudes  in  language  and  communication 


Outcomes 

Indicators 

Measures 

Comparators 

Grades 

cognitive 

achievement 

language  arts  achievement  test 
English  30  and  33  diploma  exams 

provincial  and  system 
results 

provincial  and  system 
results 

3,6&9 
12 

affective 

attitude 

School  Subjects  Attitude  Scales 
(SSAS) 

test  norms 

6,9&  12 

behavioral 

participation  in 
achievement  tests 

document  review 

system  and  provincial  rates 

3,6  &9 

participation  in  or  the  need 
to  participate  in  remedial 
programs 

document  review,  survey 

provincial  rates 

3,6  &9 

participation  in  grade  12 
diploma  courses 

document  review 

provincial  and  system  rates 

12 

course  completions 

document  review 

provincial  and  system  rates 

3,  6,9&  12 

amount  of  leisure  reading 

student  survey 

system  average 

6, 9&  12 

participation  in  related 
activities 

student  survey 

system  results 

6.9&12 

image  -  educational 
partner  satisfaction 

survey 

provincial  and  past  survey 
results 

N/A 

Figure  3:  The  Indicators,  Measures,  Comparators  and 
Grades  of  Administration  for  One  Goal 


The  process  committee  consisting  of  teachers,  administrators  and  parents  with 
representation  from  each  school  was  formed  to  identify  context,  input  and  process 
variables  that  affect  student  learning.  By  utilizing  a  number  of  system  documents 
including  the  systems'  mission  and  improvement  goals,  previous  inservice  program 
agendas,  school  system  program  and  school  evaluations,  as  well  as  a  document 
entitled  What  Works:  Research  about  Teaching  and  Learning  (US  Department  of 
Education,  1987),  the  committees  were  asked  to  identify  context,  input  and  process 
variables  that  were  relevant  in  each  system  but  were  also  identified  in  the  literature 
as  affecting  achievement 

There  was  some  initial  confusion  about  the  distinction  between  context,  input  and 
process  variables  as  well  as  a  way  of  organizing  the  variables.  As  a  result,  the 
coordinator  decided  to  use  a  categorization  adapted  from  Oakes  (1989)  that 
identified  three  sets  of  conditions  which  affect  student  achievement: 

•  access  to  learning  includes  those  variables  that  affect  students'  opportunities 
to  learn; 

•  press  for  achievement  includes  those  variables  that  encourage  students  to 
achieve  better,  and 

•  professional  teaching  conditions  are  the  variables  that  empower  or  constrain 
educators  as  they  implement  the  instructional  program. 
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It  was  felt  that  these  three  categories  would  be  more  readily  understood  by  all 
educational  partners.  Furthermore,  to  avoid  confusion  it  was  decided  to  call  them 
simply  "conditions  and  processes"  rather  than  context,  input  and  process  variables. 

Indicators  of  the  above  sets  of  conditions  and  processes  were  identified  in  four 
areas:  the  home,  classroom,  school,  and  school  system.  Figure  4  outlines  the 
condition  and  process  indicators  along  with  the  instruments  or  methods  for 
measuring  the  indicators. 


Measuring  and  Interpreting  Learning 

The  identification  of  measures  in  Figures  3  and  4  above  was  guided  by  the 
following  considerations: 

•  existing  measures  such  as  achievement  tests  and  diploma  exams  would  be 
utilized  as  much  as  possible; 

•  existing  standardized  measures  would  be  used  wherever  possible  to  enhance 
reliability  and  validity; 

•  the  measuring  devices  used  would  be  tests  and  inventories,  surveys,  document 
reviews,  self -evaluation  and  observations. 

As  a  result,  the  measurement  of  most  indicators  was  accomplished  by  utilizing 
existing  instruments  and  documents.  Three  locally  developed  instruments  were 
used  in  the  review:  the  Satisfaction  Survey,  the  Adolescent  Health  Questionnaire 
and  the  Classroom  Observation  Scales. 

The  instruments  used  to  measure  the  outcome,  condition  and  process  indicators  are 
described  below,  followed  by  a  discussion  of  the  organization  of  the  data  from  all 
instruments  to  describe  student  learning  and  to  prepare  teacher,  school,  program 
and  school  system  reports. 

Outcome  Indicators  -  The  indicators  of  student  achievement  in  the  four  areas  of 
learning  -  intellectual,  social,  personal  and  vocational  -  were  measured  by  diploma 
exams,  achievement  tests,  standardized  tests  and  inventories,  teacher  student 
evaluation  systems,  locally  developed  surveys  and  inventories,  and  document 
reviews.  The  instruments  and  documents  used  to  measure  achievement  of  the 
indicators  in  the  four  areas  of  learning  are  briefly  outlined  below. 

In  intellectual  learning,  student  achievement  in  the  core  subjects  was  measured 
by  the  provincial  achievement  tests  and  the  diploma  exams.  Achievement  in  non- 
diploma  math  and  science  courses,  in  fine  and  practical  arts,  and  in  work  experience 
was  measured  by  final  marks  awarded  by  classroom  teachers.  Achievement  in 
intellectual  curiosity  was  measured  by  the  survey  while  achievement  in  ways  of 
learning  was  measured  by  the  achievement  tests  and  diploma  exams,  portions  of 
the  Canadian  Tests  of  Basic  Skills  (1982, 1989)  and  the  Learning  Preference 
Questionnaire  (1987)  which  determines  students'  approach  to  learning. 
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Condition 

Area 

Indicator 

Measure 

3CC8SS  to  schooling 

home 

•  lamiiy  structure 

•  socioeconomic  status 

•  student  mobility 

survey,  document  review 

press  for  achievement 

home 

*  attitudes  and  actions  of  importance  to 
education 

survey 

access  to  learning 

classroom 

. 

*  instructional  presentation 

self-evaluation,  observation, 

•  classroom  environment 

survey 

4  academic  engaged  time 

•  adaptive  instruction 

•  instructional  planning 

•  grouping  practices 

•  ability  and  nature  of  class 

•  materials  and  resources 

•  parental  involvement 

press  for  achievement 

classroom 

•  teacher  expectations 

•  relevant  practice 

•  informed  feedback 

•  progress  evaluation 

•  student  recognition 

self-evaluation,  observation, 
survey 

access  to  schooling 

school 

• 

*  location  and  size 

document  review,  survey, 

•  course  offerings 

committee  study,  external  review 

•  instructional  time 

•  teacher  qualifications 

•  co-curricular  and  peer  programs 

•  school  grouping  practices 

•  parental  involvement 

•  curriculum  articulation  and  resources 

press  for  achievement 

school 

•  leadership 

•  climate 

•  academic  focus 

•  expectations 

•  feedback 

•  student  participation 

•  motivational  strategies 

•  mission 

•  parental  involvement 

•  cosmetic  care 

•  type  of  homework 

•  communication  with  public 

School  Effectiveness 
Opinionnaire,  external  validation, 
survey 

professional  teaching 

school 

•  class  size 

document  review,  survey, 

conditions 

•  teacher  planning  time 

•  teacher  involvement  in  decision  making 

•  teacher  autonomy  and  flexibility 

•  clerical  and  paraprofessional  support 

•  teacher  attitude  and  morale 

School  Effectiveness 
Opinionnaire 

access  to  schooling 

school  system 

•  location,  size  and  assessment  base 

•  public  satisfaction  with  education 

•  parental  involvement  in  decision  making 

•  professional  development 

•  support  for  curriculum  enhancement 

document  review,  survey 

press  for  achievement 

school  system 

•  mission,  goals  and  actions  for  improvement 

•  high  expectations  for  employee  performance 
and  student  achievement 

•  recognition  of  employees  and  students 

•  monitor  student  achievement 

•  mnnitnr  instructional  nrrinram 
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•  communication  with  public 

document  review,  survey 

professional  teaching 

school  system 

•  staff  involvement  in  system  decision  making 

document  review,  survey 

conditions 

•  school  flexibility  and  autonomy 

Figure  4:  Indicators  of  Conditions  and  Processes  Affecting  Student  Achievement 
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Participation  rates  for  high  school  courses,  the  percentage  of  students  taking  each 
course,  was  calculated  using  data  from  two  documents,  the  School  Program  Plan 
and  the  Summary  of  Enrollment  by  Age,  Grade  and  Sex.  Provincial  rates  were 
calculated  by  using  data  on  the  numbers  of  course  participants  and  total  grade 
enrollments  provided  by  Alberta  Education  Information  Services.  Participation 
rates  for  students  in  core  subjects  in  grades  3, 6  and  9  were  calculated  from  the 
jurisdiction  reports  for  achievement  tests  by  dividing  the  number  of  students  writing 
by  the  total  grade  enrollment. 

Completion  rates  for  high  school  courses  were  determined  by  using  information 
from  the  School  Program  Plan,  the  jurisdiction  diploma  exam  report,  and  in  the 
case  of  nondiploma  exam  courses,  the  summary  reports  of  final  marks.  Completion 
rates  for  grades  3,  6  and  9  were  calculated  from  the  jurisdiction  reports  by  dividing 
the  number  achieving  the  acceptable  standard  by  the  number  eligible  to  write  the 
test. 

Student  attitudes  in  core  subjects  were  measured  by  the  School  Subjects  Attitude 
Scales  (1983)  while  student  and  partner  satisfaction  with  learning  and  student 
participation  in  related  activities  was  measured  by  the  Satisfaction  Survey. 

In  social  learning,  development  of  personal  attributes  and  social  skills,  as  well  as 
involvement  in  promotional  activities,  was  measured  by  the  survey  and  review  of 
documents. 

In  personal  learning,  physical  well-being  was  measured  by  participation  in  senior 
high  physical  education  courses,  by  the  Canada  Fitness  Award  (1984),  the 
Satisfaction  Survey  and  through  the  Adolescent  Health  Questionnaire  which 
measured  four  components  of  physical  well-being:  eating  habits,  involvement 
in  activities,  substance  use  and  sexuality. 

The  indicators  of  emotional  well-being  were  measured  by  a  standardized 
instrument,  the  survey,  review  of  documents  and  the  locally-developed  health 
inventory.  Student's  self-concept  was  measured  by  the  Piers-Harris  Children's 
Self-Concept  Scale  (1984),  a  scale  that  measures  overall  concept  as  well  as  six 
subscales:  behavior,  intellectual  and  school  status,  physical  appearance  and 
attributes,  anxiety,  popularity,  and  happiness  and  satisfaction.  Through  the  survey, 
student,  parent  and  teacher  satisfaction  with  student  development  of  desirable 
characteristics,  such  as  self-discipline  and  goal  orientation,  was  measured. 

Condition  and  Process  Indicators  -  As  indicated  in  Figure  4,  the  condition  and 
process  indicators  were  measured  by  five  means:  the  satisfaction  survey,  document 
review,  a  standardized  instrument,  the  Classroom  Observation  Scales  and 
committee  study.  Use  is  described  in  four  areas:  the  home,  classroom,  school, 
and  school  system. 

In  the  home,  the  survey  was  used  to  measure  family  structure  and  attitudes  whereas 
document  review  was  utilized  for  socioeconomic  status  and  student  mobility. 

In  the  classroom,  the  survey  was  used  to  measure  some  indicators  like  grouping 
practices  and  student  recognition.  The  locally  developed  Classroom  Observation 
Scales  were  used  for  all  indicators  related  to  teaching  and  learning,  such  as 
instructional  presentation  and  academic  engaged  time.  The  process  for  assessing 
these  indicators  involved  teacher  self -evaluation,  observation  and  external 
evaluation  by  school  administrators  and  external  validation  by  system  personnel. 
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In  the  school,  using  the  School  Program  Plan,  teacher  and  school  timetables,  and 
the  report  on  teacher  qualifications,  indicators  such  as  course  offerings,  instruction 
time,  class  size  and  teacher  planning  time  were  measured.  The  survey  measured  a 
number  of  indicators  like  grouping  practices,  parental  involvement  and  teacher 
involvement  in  decision  making.  Through  committee  study  followed  by  external 
review,  curriculum  articulation  in  the  core  subjects  was  measured.  The  School 
Effectiveness  Opinionnaire  (1989)  measured  ten  school  effectiveness  indicators: 
leadership,  academic  focus,  climate,  expectations,  feedback,  student  participation, 
motivational  strategies,  mission,  parental  involvement  and  cosmetic  care.  Following 
compilation  of  the  opinionnaire  results,  an  external  validation  team  reviewed  the 
data  for  a  number  of  indicators. 

In  the  school  system,  a  review  of  Education  in  Alberta:  Facts  and  Figures  1991, 
the  jurisdiction  policy  handbooks  and  system  planning  documents  were  used  to 
assess  a  number  of  system  indicators  like  assessment  base,  parent  and  staff 
involvement  in  decisions  and  support  for  curriculum  enrichment.  The  survey  was 
used  to  augment,  confirm  or  dispute  policy  directions  for  the  same  indicators. 


Organization  of  the  Findings  for  Reporting 

Since  the  focus  of  the  school  system  review  process  was  on  student  learning  and 
the  conditions  and  processes  contributing  to  learning,  the  findings  of  the  review 
have  been  organized  and  reported  by  student  achievement  of  each  goal  followed  by 
the  relevant  findings  on  the  conditions  and  process.  In  this  way,  a  comprehensive 
description,  including  overall  achievement,  achievement  of  major  components, 
participation  and  completion  rates,  attitude,  participation  in  related  activities  and 
student  and  partner  satisfaction  with  learning  is  given  for  each  goal.  A  sample  of  the 
description  of  student  learning  is  included  in  the  section  on  "Findings  on  Student 
Learning". 

Information  on  conditions  and  processes  affecting  learning  in  mathematics  has  not 
been  included  in  this  description  since  the  assembly  and  assessment  of  the  data  by 
the  process  committees  have  not  been  completed  at  the  time  of  writing.  Determination 
of  the  conditions  and  processes  affecting  each  goal  will  be  decided  by  the  collective 
judgment  of  the  committee  rather  than  by  statistical  means. 

Nature  of  Reports  -  Sufficient  information  was  gathered  through  all  of  the 
measures  to  produce  a  variety  of  individual,  program,  school  and  jurisdiction 
reports.  The  relationship  of  the  information  gathered  on  all  indicators  to  the  various 
reports  is  as  follows: 

1 .  at  the  classroom  level,  individual  confidential  teacher  reports  were  written  using 
the  data  on  classroom  indicators; 

2 .  at  the  program  level,  core  program  reports  can  be  written  using  the  student 
outcome  indicators,  the  aggregated  information  from  the  teacher  evaluation 
reports  and  selected  home  and  school  indicators; 

3 .  at  the  school  level,  a  school  report  was  written  on  each  goal  of  learning, 
including  the  information  on  all  home  and  school  indicators,  student  outcome 
and  classroom  indicators; 
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4.  at  the  jurisdiction  level,  a  system  report  was  written  using  the  information  from 
all  school  system  indicators  as  well  as  home,  school,  outcome  and  classroom 
indicators. 


The  Interpretative  Process  -  Following  measurement  of  all  indicators,  student 
achievement  on  the  goals  of  learning  will  be  evaluated  through  an  interpretative 
process.  An  interpretative  team,  representing  all  educational  partners,  will  be 
formed  for  each  domain  of  learning  (intellectual,  social,  personal  and  vocational) 
in  each  jurisdiction.  Each  team's  task  will  be  to  review  the  results  of  each  goal  of 
learning,  and  decide  whether  the  results  were  excellent,  very  good,  satisfactory 
or  need  improving.  To  make  this  evaluative  judgment  the  team  members  will 
familiarize  themselves  with  the  instruments  by  completing  each  instrument,  by  prior 
establishment  of  desirable  and  acceptable  levels  of  achievement,  by  determining  the 
relationship  of  outcomes  to  the  comparators  and  also  the  interrelationship  of  the 
indicators,  and  by  determining  the  conditions  and  processes  affecting  the  outcomes. 
Finally,  after  having  made  a  judgment  about  the  results  on  each  goal,  they  will 
determine  strengths  and  weaknesses  and  identify  strategies  for  improvement. 

Following  the  interpretations  in  each  domain  of  learning,  the  steering  committee 
will  review  the  results  of  each  interpretation  committee  to  insure  consistency. 
Interpretative  teams  in  each  school  will  interpret  school  results  through  a  similar 
process. 


A  Plan  for  the  Future 

To  ensure  that  there  is  a  follow-up  to  the  evaluation,  a  modified  strategic  planning 
process  has  been  incorporated  as  an  integral  part  of  the  school  system  review.  The 
general  role  of  the  planning  team,  representing  all  partners,  will  be  to  take  the 
strengths  and  weaknesses  identified  in  the  interpretation  process  as  well  as  other 
information  and  reorder  curricular,  human  and  physical  resources  to  facilitate 
improvements  in  learning  in  weak  areas  and  to  maintain  strong  areas  of  learning. 
Specifically,  the  planning  team  will: 

•  review  the  mission,  beliefs  and  goals  of  learning; 

•  review  pertinent  information  internally  and  externally  that  impact  on  the  system, 
including  the  results  of  the  school  system  review; 

•  identify  critical  issues  and  concerns; 

•  establish  improvement  goals  and  strategies  to  achieve  the  goals. 

Following  the  completion  of  these  activities,  action  plans  to  implement  the  strategies 
will  be  developed  by  committees  representing  all  partners.  At  this  point,  board 
consideration  of  the  plan  including  costing  will  occur. 
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The  School  System  Review 


The  study  resulted  in  the  development  of  a  comprehensive  school  system  review 
process  based  on  student  learning  and  the  conditions  and  processes  affecting 
learning.  It  has  developed  a  way  of  measuring  student  learning  with  a  system  of 
indicators  on  a  comprehensive  statement  of  goals  of  learning,  of  measuring  a  full 
range  of  conditions  and  processes  affecting  student  learning,  and  relating  the 
conditions  and  processes  to  student  learning.  Sufficient  data  are  produced  by  the 
various  instruments  and  processes  to  generate  individual  confidential  teacher 
reports,  core  program  reports,  school  and  school  system  reports. 

The  review  system  consists  of  four  related  components: 

•  a  handbook  of  procedures  for  implementing  the  school  system  review  process; 

•  supplementary  handbooks  on  planning,  communicating  the  results  and 
providing  inservice  to  staff; 

♦  reports  on  the  detailed  findings  for  a  number  of  the  instruments  and  procedures; 

♦  system  and  school  reports  on  the  findings  of  the  school  system  review  process. 

Each  of  these  components  is  described  forthwith. 


Handbook  of  Procedures  for  Implementing  the  School  System  Review  Process 

This  document  is  a  complete  guide  for  implementing  the  school  system  review.  It 
describes  the  process  including  timelines  and  personnel  responsibilities,  provides  a 
detailed  description  for  measuring,  analyzing  and  assessing  the  data  for  all  of  the 
indicators  and  gives  the  findings  for  each  of  the  measures.  The  appendices  provide 
a  detailed  guide  to  the  adniinistration  of  each  instrument,  as  well  as  a  copy  of  each 
instrument  developed  for  this  study. 

Overview  of  the  Review  Process  -  The  handbook  gives  an  overview  of  the  review 
process  and  then  provides  a  brief  description  of  each  of  the  components  in  the 
process:  the  goals  of  learning,  the  outcome  indicators,  measures  and  comparators, 
the  condition  and  process  indicators  and  their  measures.  The  organization  of 
findings  by  goal  for  a  final  report  is  described  and  the  procedures  for  generating 
teacher,  program,  school  and  school  system  reports  is  given.  Personnel 
responsibilities  and  timelines  for  adniinistration  are  included.  Finally,  it  describes 
a  process  for  interpreting  the  results  and  developing  a  plan  for  the  future. 

Procedures  for  Measuring,  Analyzing  and  Assessing  the  Data  for  All  Indicators  - 
The  largest  part  of  the  handbook  is  devoted  to  describing  how  each  indicator  is 
measured,  how  the  information  is  analyzed  and  displayed,  and  how  the  data  are 
assessed  and  summarized  for  the  interpretation  committee.  Finally,  a  summary  of 
the  data  that  were  collected  on  each  of  the  indicators  in  1991-92  is  displayed. 

The  Appendices  -  Further  detail  is  given  on  the  administration  of  the  more  complex 
measures  in  the  review  process.  Generally,  a  document  entitled  "Guide  to  the 
Administration  of  the ..."  is  provided  which  gives  the  complete  administration 
procedures  for  the  instrument.  Of  particular  interest  in  the  appendix  is  a  "Survey 
Resource  Manual",  developed  by  a  reputable  market  consultant,  which  describes 
the  appropriate  methods  for  developing  and  administering  a  survey. 
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Supplementary  Handbooks 

Supplementary  handbooks  have  been  prepared  describing  in  detail  the  planning 
process,  the  communication  plan  and  the  staff  inservice  plan  on  the  findings  of 
the  review  process. 


Reports  on  the  Detailed  Findings  for  Some  Instruments 

Separate  reports  provide  the  detailed  findings  for  a  number  of  the  instruments 
administered  in  1991-92.  They  include: 

•  School  Effectiveness  Opinionnaire  (Renihan  &  Sackney,  1989); 

•  Satisfaction  Survey; 

•  Learning  Preference  Questionnaire  (Australian  Council  for  Educational 
Research,  1987); 

•  Piers-Harris  Children's  Self-Concept  Scales  (Piers,  1984); 

•  School  Subjects  Attitude  Scales  (Nyberg  &  Clarke,  1983) 

•  Attitudes  Toward  the  World  of  Work  (Maguire,  Romaniuk,  &  MacRury, 
1983). 


System  and  School  Reports  on  the  Findings  of  the  School  System  Review  Process 

The  system  findings  on  all  of  the  indicators  have  been  reorganized  into  a  format 
based  on  each  goal  of  learning.  After  a  brief  description  of  the  school  system 
review  process,  a  full  description  of  student  learning  with  data  from  the  appropriate 
instruments  is  given  followed  by  the  condition  and  process  indicators  that  are 
contributing  to  learning  and  those  that  are  not.  In  the  discussion,  the  conclusion  on 
student  learning  for  the  goal  is  given  and  plans  for  maintaining  or  strengthening 
learning  are  detailed.  This  report  tends  to  be  brief,  relating  all  of  the  findings  to  one 
focal  point,  student  learning,  thus  making  sense  of  all  of  the  data  that  have  been 
collected.  It  is  a  comprehensive  report  to  the  public  on  the  findings  of  the  school 
system  review. 

The  findings  for  each  school  on  all  indicators  were  also  organized  by  goals  of 
learning  and  a  report  was  written  on  the  findings  for  each  school.  These  reports  are 
for  the  use  of  each  school  community. 


The  Findings 

The  study  was  designed  to  produce  two  kinds  of  findings:  the  feasibility  of  the 
school  system  review  process,  and  the  state  of  student  learning  and  the  contributing 
conditions  and  processes.  The  findings  on  the  process  have  resulted  from 
professional  judgments  as  the  process  was  developed  and  implemented,  whereas 
the  findings  on  student  learning  are  the  data  produced  by  the  various  instruments. 
This  section  gives  the  findings  on  the  feasibility  of  the  process  and  the  system 
findings  on  a  sample  of  the  goals  of  learning. 
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The  Process 


Overall  Design  -  The  educational  partners  and  the  technical  advisors  supported  the 
review  process  as  a  sound  one  including  the  overall  design,  based  on  the  goals  of 
student  learning  with  a  system  of  indicators  of  achievement,  the  instruments  and 
processes,  a  way  of  interpreting  the  findings,  and  a  planning  process  to  utilize  the 
findings. 

The  Goals  of  Learning  -  The  goals  of  learning  (intellectual,  social,  personal  and 
vocational)  were  found  to  provide  an  overall  balanced  statement  of  goals.  Within  a 
number  of  domains  of  learning,  there  was  also  a  balance  of  goals  that  included  the 
development  of  skills,  knowledge  and  attitudes. 

Indicators,  Measures  and  Comparators  -  In  general  it  was  found  that  there  was  a 
comprehensive  set  of  indicators  and  appropriate  measures  and  comparators  to 
measure  and  interpret  learning  on  most  goals.  There  were  some  concerns  with  the 
measures  for  intellectual  curiosity  and  the  desire  for  lifelong  learning,  the  measures 
for  social  learning  and  the  comparators  for  learning  in  the  intellectual  domain. 
Recommendations  for  alleviating  these  concerns  are  made  later  in  this  summary. 

The  findings  on  the  usefulness  of  the  major  data  collection  instruments  and 
procedures  follow. 

The  Satisfaction  Survey  produced  data  on  a  large  number  of  indicators  since  the 
systems  opted  for  a  longer  mail-out  version  rather  than  a  shorter  telephone  survey. 
Its  length  and  complexity  meant  that  returns  were  slow  and  that  the  analysis  took 
more  time  and  resources  than  anticipated.  In  fact  the  extended  analysis  stretched  the 
resources  for  the  analysis  of  other  instruments  and  processes. 

The  School  Effectiveness  Opinionnaire  produced  interesting  data  on  school 
indicators.  Its  administration  and  analysis  were  straightforward  and  efficient. 

Use  of  the  diploma  exam  and  achievement  test  jurisdiction  reports,  the  School 
Program  Plan,  the  Summary  of  Enrollment  by  Age,  Grade  and  Sex  as  well  as 
comparable  data  from  Information  Services,  made  it  possible  to  calculate  overall 
achievement,  participation  rates  and  completion  rates  for  these  subjects. 
Unfortunately,  achievement  on  major  course  components  of  the  diploma  exams, 
such  as  the  cognitive  levels  of  knowledge,  comprehension  and  higher  level  skills, 
cannot  be  determined  in  the  same  units  as  overall  achievement.  Overall  achievement 
is  given  in  percentage  achieving  the  acceptable  standard  whereas  achievement  of 
major  components  is  given  as  a  raw  score.  The  result  is  a  loss  in  interpretative 
power. 

Student  achievement  in  nondiploma  exam  courses  was  measured  by  analyzing 
summaries  of  teacher-awarded  final  marks.  The  collection  procedures  unfortunately 
were  not  as  well  defined  as  they  should  have  been  causing  delays  in  receiving  the 
data  and  considerably  more  effort  in  analyzing  the  results. 

The  use  of  the  Classroom  Observation  Scales  to  measure  classroom  indicators  was 
conducted  without  difficulty.  Although  the  coordinator  had  concerns  initially  about 
the  time  to  carry  out  the  evaluations,  the  school  administrators  readily  accomplished 
the  task  since  they  would  have  devoted  a  similar  amount  of  time  to  evaluation  under 
"normal"  circumstances.  The  inservice,  structure  and  process  were  appreciated  by 
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those  involved  in  the  evaluations.  Use  of  self-evaluation  by  teachers,  followed  by 
external  evaluation  by  school  administrators  and  validation  by  system  personnel 
provided  the  appropriate  balance  of  participation  from  each  level. 

All  school  staffs  completed  a  review  of  the  curriculum  articulation  and  sequencing 
in  each  of  the  core  subjects.  The  outlined  procedures  as  well  as  the  program  review 
guides  provided  the  necessary  structure  to  complete  the  review.  The  validation  by 
system  personnel  in  one  instance  and  regional  office/system  personnel  in  the  other 
instance  were  able  to  validate  the  internal  findings  by  using  the  outlined  procedures. 

The  implementation  of  the  data  collection  procedures  for  all  of  the  instruments  in 
the  same  school  year  resulted  in  demands  on  classroom  and  personnel  time. 
Fortunately  each  of  the  systems  canceled  all  other  evaluation  activities  for  the  1991- 
92  school  year  making  time  available  for  implementing  the  review  process.  By 
using  existing  measures  wherever  possible  and  by  administering  the  instruments  to 
different  grade  levels,  the  time  demands  in  any  one  set  of  classrooms  were 
rrunimized.  The  impact  on  teacher  time  was  found  to  be  little  different  from  the 
usual  since  surveys  and  program  reviews  had  been  the  norm  in  both  systems. 
Teacher  involvement  in  self-evaluation  did  impact  on  teacher's  time,  but  their 
appreciation  in  being  more  involved  in  evaluation  more  than  compensated  for  the 
time  spent 

Interpreting  Learning  -  The  process  of  interpreting  the  findings  whereby  the 
interrelationship  of  indicators,  the  relationship  of  results  to  comparators  and  the 
contribution  of  conditions  and  processes  to  make  a  judgment  about  the  value  of  the 
results  was  found  to  be  an  appropriate  process.  This  was  confirmed  by  the  technical 
advisors.  The  process  to  date  has  not  been  implemented  so  it  is  not  possible  to 
comment  on  the  workability  of  process. 

A  Plan  for  the  Future  -  This  plan  has  been  modeled  after  a  traditional  strategic 
planning  process  with  the  usual  components.  Overall  the  process  is  sound  with 
an  added  advantage  of  being  based  on  the  strengths  and  weaknesses  of  student 
learning  and  the  contributing  conditions  and  processes.  Thus,  the  plan  for  the  future 
begins  with  the  focal  point  of  "student  learning"  and  then  identifies  conditions  and 
processes  that  need  to  be  changed  to  improve  achievement  The  planning  has  not 
yet  taken  place  so  the  feasibility  of  this  process  cannot  be  assessed. 


Findings  on  Student  Learning 

To  show  the  nature  of  the  findings,  student  learning  on  a  sample  of  goals  is 
described.  Goals  have  been  selected  where  different  measures  have  been  used, 
learning  is  described  in  mathematics,  emotional  well-being,  and  vocational 
learning. 

Unfortunately,  the  findings  on  the  conditions  and  processes  cannot  be  included 
because  the  process  committees  have  not  completed  assessing  the  data  for  inclusion 
with  the  outcome  data. 

Mathematics  -  Table  1  presents  the  indicators  of  student  learning  in  grade  9  for 
the  County  of  Lacombe.  Learning  is  described  by  achievement,  participation, 
completion,  attitudes,  participation  in  related  activities  and  partner  satisfaction. 
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Table  1 


Student  Learning  in  Grade  9  Mathematics, 
County  of  Lacombe,  1991-92 


Grade  Indicators 


Jurisdiction  Province 

N  (%)       N  (%) 


achievement  * 

•  total  test 

-  acceptable  standard 

-  standard  of  excellence 

•  problem  solving 

-  acceptable  standard 

-  standard  of  excellence 

•  subject  matter 

-  acceptable  standard 

-  standard  of  excellence 

participation  2 
completion  3 


242 


27,888 


64.5 

67.4 

6.6 

8.9 

59.5 

64.1 

8.3 

10.7 

76.4 

75.6 

9.5 

12.7 

88.0 

81.1 

61.7 

61.3 

attitude4 

•  evaluation  scale 

•  usefulness  scale 

•  difficulty  scale 


244 


participation  in  related  activities  5  244 


educational  partner  satisfaction  6 

•  students 

•  parents 

•  teachers 


247 
113 
71 


mean  24.0 
mean  32.9 
mean  20.1 

0.0 


79.5 
77.2 
70.6 


1  Achievement  as  determined  by  the  June  1992  provincial  achievement  test  where  85%  of 
students  are  expected  to  achieve  the  acceptable  standard  and  15%  are  expected  to  achieve  the 
standard  of  excellence. 

2  Participation  is  the  percentage  of  students  writing  the  test. 

3  Completion  is  the  percentage  of  students  eligible  to  write  the  test  that  achieved  the  acceptable 
standard. 

4  Attitude  was  measured  by  the  School  Subjects  Attitude  Scales  at  midyear;  a  mean  of  24 
represents  a  neutral  score  whereas  a  mean  above  24  represents  a  positive  attitude  toward  a 
subject  and  one  below  24  a  negative  attitude. 

5  A  per  capita  participation  rate  was  determined  by  dividing  the  number  of  instances  of 
participation  by  the  grade  enrollment. 

6  The  percentage  of  partner  satisfaction  with  learning  mathematics  was  determined  by  the 
locally  developed  Satisfaction  Survey. 
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Emotional  Well-being  -  Table  2  presents  the  state  of  students'  emotional  well-being 
in  the  County  of  Lacombe  for  grades  6, 9  and  12  in  1991-92.  Emotional  well-being 
is  described  by  three  indicators:  self-concept,  partner  satisfaction  and  evidence  of 
vandalism,  suspension  and  expulsion. 

Table  2 

Student  Emotional  Well-Being,  County  of  Lacombe 
Grades  6,  9  and  12,  1991-92 


Indicators                         Grade  6  Grade  9  Grade  12 

Female     Male  Female    Male  Female  Male 
N=157   N=158  N=134   N=135  N=84  N=104 
 (PR)      (PR)  (PR)      (PR)  (PR)  (PR) 


self-concept 


•  total  test  69 

•  behavior  64 

•  intellectual  &  school  status  59 

•  physical  appearance  56 
&  attributes 

•  anxiety  55 

•  popularity  30 

•  happiness  &  satisfaction  57 


68 

60 

39 

65 

56 

43 

49 

26 

63 

39 

51 

58 

34 

55 

35 

62 

53 

39 

61 

58 

66 

50 

51 

51 

60 

39 

44 

26 

46 

44 

60 

53 

40 

60 

54 

Students  Parents  Teachers 

Grade 


satisfaction  with  learning  2 

•  self-concept 

•  self-discipline 

•  responsibility 

•  goal  orientation 

•  pursuit  of  excellence 

•  adaptability 


suspension  &  expulsion  3 
vandalism  4 


6 

9 

12 

6 

9 

12 

6 

9  12 

300 

247 

223 

336 

113 

112 

101 

71  61 

(%) 

(%) 

(*) 

73 

82 

83 

83 

73 

79 

89 

89  80 

72 

89 

75 

88 

64 

71 

84 

62  58 

78 

90 

88 

83 

77 

76 

82 

65  64 

73 

85 

75 

77 

72 

65 

78 

60  65 

71 

84 

75 

84 

62 

60 

56  56 

69 

91 

80 

91 

79 

75 

91 

92  71 

Gr. 

1-6 

Gr.  7-9 

Gr.  10-12 

1 

.5% 

4.1% 

1.0%  [ 

.2% 

1  Percentile  rank  of  group  means  for  self-concept  determined  by  the  Piers-Harris  Children's 
Self-Concept  Scale. 

2  The  percentage  of  satisfaction  with  learning  desirable  characteristics  related  to  emotional  well-being 
as  determined  by  the  locally  developed  Satisfaction  Survey. 

3  The  number  of  instances  of  suspensions  and  expulsions  divided  by  the  enrollment  of  each  group. 

4  The  number  of  instances  of  vandalism  divided  by  the  total  enrollment  in  grades  1  to  12. 
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Table  3 

Vocational  Learning  in  Rocky  Mountain  School  Division, 
Grades  9  to  12,  1991-92 


Grades    Indicators  Jurisdiction  Province 

(N=105)  (N=l  1,499) 

 m  is  

10-12        work  experience 

•  achievement1  88.0  87.3 

•  participation  2                                             13.1  10.1 
 *  completion3  1^1  

12  Students      Parents  Teachers 

(N=179)         (N=73)  (N=37) 

 (%)  (%)  m 


partner  satisfaction  4 

•  prompt 

81.0 

73.6 

57.9 

•  neat 

87.9 

80.2 

77.4 

•  industrious 

67.2 

71.1 

55.4 

•  persevering 

84.4 

77.5 

56.4 

•  work  habits,  attitudes,  skills  and  knowledge 

76.3 

69.7 

57.6 

9  &  12  Grade  9  Grade  12 

Female           Male            Female  Male 
(N=91)         (N=104)           (N=93)  (N=79) 
 (PR)  (PR)  £PR)  (PR) 


attitudes 


preparation 

39 

52 

38 

46 

interest 

34 

43 

20 

27 

diligence 

36 

38 

44 

36 

laziness 

64 

69 

57 

50 

job  security 

73 

68 

68 

62 

positive  employer 

52 

51 

38 

32 

characteristics 

independence 

67 

63 

41 

53 

money 

60 

50 

38 

49 

ambition 

71 

62 

47 

56 

locus  of  control 

75 

71 

71 

67 

confidence 

28 

39 

23 

22 

negative  employer 

79 

74 

70 

62 

characteristics 

social  relations 

44 

36 

27 

33 

unemployment 

66 

60 

76 

66 

general  attitudes 

35 

58 

60 

46 

1  Percentage  of  students  achieving  50%  or  higher  in  work  experience. 

2  Percentage  of  enrolled  high  school  students  taking  the  course. 

3  Percentage  of  students  that  initially  enrolled  in  work  experience  and  achieved  50%  or  better. 

4  Percentage  of  satisfaction  with  learning  desirable  characteristics  related  to  work  as  measured  by 
the  locally  developed  Satisfaction  Survey. 

5  Percentile  rank  of  group  means  for  student  attitudes  determined  by  the  Attitudes  Toward  the 
World  of  Work. 
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Vocational  Learning  -  Table  3  presents  student  learning  in  the  Rocky  Mountain 
School  Division  for  grades  9  to  12  in  1991-92.  Vocational  learning  is  described 
by  achievement,  participation  and  completion  in  work  experience,  by  partner 
satisfaction  and  by  attitudes  to  work. 


Conclusions 

The  development  and  implementation  of  the  school  system  review  has  resulted  in 
significant  conclusions.  First,  the  process  is  pedagogically  sound  because  the  focal 
point  of  the  review  is  student  learning.  Second,  although  most  evaluation  systems 
heretofore  have  focused  on  the  process  of  education,  the  school  system  review 
more  appropriately  focuses  on  student  learning  and  relates  conditions  and  processes 
to  achievement 

Implementation  of  the  school  system  review  has  been  very  demanding,  but  it 
was  found  to  be  feasible  in  terms  of  clarity  of  processes,  time  and  personnel  for 
implementation  and  the  minimal  time  it  takes  from  classroom  learning.  Although 
there  were  significant  time  demands  on  school  administrators  and  staffs,  the  nature 
of  the  involvement  resulted  in  a  greater  level  of  teamwork  and  enhanced  sense  of 
ownership  of  the  evaluation  and  its  results. 

The  findings  from  the  development  and  implementation  of  the  process  show  that 
a  comprehensive  system  of  indicators  to  describe  learning  and  the  contributing 
processes  is  possible.  The  study  has  identified  a  significant  set  of  indicators  of 
learning  and  the  conditions  and  processes  affecting  learning  that  are  measurable 
and,  when  seen  as  a  system,  tell  a  comprehensive  story  about  student  learning. 

The  review  process  was  very  effective  in  maximizing  the  use  of  the  data  to  generate 
teacher,  school,  program  and  school  system  reports. 

Overall,  the  development  and  implementation  of  the  school  system  review  has  been 
very  successful  resulting  in  a  sound  and  feasible  review  process  in  the  two 
jurisdictions. 


Implications 

The  findings  of  this  study  have  implications  for  education  in  general  and  specific 
implications  for  the  two  participating  systems,  other  jurisdictions  and  the  province. 

The  design  and  findings  clearly  imply  that  the  focus  of  an  evaluation  system  in 
educational  institutions  should  be  on  student  learning.  Although  the  "process" 
evaluations  that  have  occurred  in  the  past  are  designed  to  improve  the  process  and 
hence  learning,  the  time  has  come  to  relate  process  evaluations  and  student  learning 
in  a  more  direct  way. 

Attempts  to  explain  student  achievement  or  the  health  of  a  school  system  or 
education  in  the  province  on  the  basis  of  one  indicator  are  clearly  inadequate.  Use 
of  other  indicators  along  with  achievement  provides  a  better  picture  of  achievement. 
The  review  process  shows  that  it  is  possible  to  measure  affective  and  behavioral 
goals  and,  therefore,  this  study  has  implications  for  school  systems  to  evaluate 
more  than  cognitive  goals. 
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Evaluation  activities  should  be  seen  as  a  system  of  interrelated  components  which 
can  produce  a  variety  of  outcomes.  Typically,  evaluations  are  one-purpose 
activities,  for  example,  teacher  evaluations,  program  evaluations,  review  of  diploma 
results  and  satisfaction  surveys.  Not  only  is  this  an  inefficient  way  of  conducting 
evaluations,  but  it  does  not  reveal  the  interrelationships  of  the  findings  from  the 
various  evaluation  activities.  Through  the  review  process,  a  variety  of  reports  can 
be  produced  simultaneously  that  show  the  relationship  of  conditions  and  processes 
to  outcomes. 

The  design  and  findings  show  an  appropriate  emphasis  on  evaluation,  planning  and 
implementing  improvements.  While  the  review  process  requires  a  conceited  effort 
on  evaluation  for  one  year,  it  frees  up  four  years  for  planning  and  implementing 
improvement.  Overall  it  will  mean  mat  system  and  school  administrators  and  staff 
will  be  able  to  devote  more  time  to  improving  learning.  These  improvements  will 
result  from  more  comprehensive  data  and,  therefore,  will  stand  a  better  chance  of 
improving  learning. 

The  review  will  provide  more  honest  information  for  accounting  to  the  public. 
Current  practice  of  reporting  one  indicator  presents  only  a  part  of  the  situation  to  the 
public.  Reporting  a  system  of  indicators  tells  a  more  comprehensive  story. 

For  other  school  jurisdictions,  a  sound  and  feasible  school  system  review  process 
is  available  for  use.  It  is  not  suggested  that  the  full  review  be  simply  implemented 
as  is.  What  is  suggested  is  that  the  five  components  are  integral  to  the  process  and 
should  be  a  part  of  any  review.  If  a  jurisdiction  wanted  to  implement  a  system 
review,  it  would  be  necessary  to  study  each  of  the  components  to  determine 
whether  each  would  satisfy  local  needs.  For  example,  the  following  questions 
could  guide  a  review  of  the  components.  Are  the  goals  of  learning  appropriate? 
Are  indicators  of  learning  appropriate?  Are  there  better  ways  of  measuring  the 
indicators?  In  this  way,  the  overall  review  process  could  be  tailored  to  address  local 
circumstances. 

For  the  province,  the  study  shows  that  it  is  possible,  even  within  current  evaluation 
programs  and  information  systems,  to  report  on  the  health  of  the  education  system 
on  a  simple  set  of  indicators.  It  is  possible  right  now  to  report  on  participation  and 
completion  rates  along  with  achievement  on  the  diploma  exams.  It  is  also  possible 
with  data  provided  by  Information  Services  to  provide  provincial  results  on 
achievement,  participation  and  completion  rates  in  all  senior  high  courses.  Using 
what  is  currently  available,  the  province  could  provide  better  provincial  data  for  its 
own  planning  and  for  public  accountability. 


Recommendations 

To  improve  the  school  system  review  process,  modification  should  be  made  to 
some  of  the  instruments  for  measuring  the  indicators. 

•    The  satisfaction  survey  should  be  shortened  and  the  information  should  be 
collected  through  a  telephone  survey.  An  investigation  of  the  costs  shows  that  a 
telephone  survey  is  little  if  any  more  expensive  and  a  considerably  more 
efficient  way  of  collecting  data. 
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•  Other  ways  of  measuring  intellectual  curiosity  and  the  desire  for  lifelong 
learning  should  be  investigated.  The  three  current  indicators,  participation  in 
related  activities,  educational  partner  satisfaction  and  involvement  of  graduates 
in  further  education,  are  insufficient  to  describe  learning. 

•  The  data  collection  and  analysis  procedures  for  learning  in  the  nondiploma 
subjects  should  be  further  developed. 

•  The  technology  use  survey  had  to  be  postponed  because  of  the  strike.  It  should 
be  developed  for  future  use  in  school  system  reviews. 

•  The  use  of  the  Fort  McMurray  Public  District's  Social  Skills  Diagnostic  Screen 
and  the  processes  for  measuring  social  learning  should  be  investigated.  This 
would  add  a  dimension  of  achievement  in  social  learning  to  our  current 
indicators  of  partner  satisfaction,  achievement  of  desirable  characteristics  and 
participation  in  related  activities. 

•  The  use  of  exemplary  pieces  of  student  work  (paintings,  videotapes,  writing)  in 
various  subjects  should  be  investigated  for  use  in  interpreting  student  learning 
as  recommended  by  the  technical  advisors. 

•  The  two  school  jurisdictions  should  monitor  their  activities  in  the  four 
intervening  years  to  maintain  the  integrity  of  the  review  process. 

It  is  recommended  to  Alberta  Education  that  all  achievement  test  and  diploma  exam 
results  be  reported  in  the  jurisdiction  summaries  in  the  same  units.  If  the  results  on 
the  major  components  were  reported  in  the  same  units  as  overall  achievement, 
namely  the  percentage  achieving  the  acceptable  standard  and  the  standard  of 
excellence,  the  results  would  be  more  meaningful. 

Further  it  is  recommended  that  Alberta  Education  make  data  available  to  calculate 
participation  and  completion  rates. 

Lastly,  it  is  recommended  that  Alberta  Education  report  participation  and  completion 
rates  in  jurisdiction  reports  along  with  achievement  on  diploma  exams,  and  finalize 
a  set  of  provincial  indicators  as  a  result  of  the  Educational  Quality  Indicators 
initiative. 


Follow-up 

The  intent  is  to  carry  out  the  school  system  review  process  every  five  years  with  the 
intervening  years  being  devoted  to  implementing  the  improvement  plans.  To  insure 
that  the  integrity  of  the  review  process  is  maintained,  the  systems  should  monitor 
the  system  activities  in  the  four  intervening  years  to  insure  that  the  main  thrust  is 
improvement  of  learning  and  that  "results  on  single  indicators  or  single  incidents" 
do  not  cause  a  reversion  to  former  evaluation  activities.  The  monitoring  should,  on 
the  other  hand,  insure  that  required  evaluations  are  identified  and  conducted.  For 
example,  in  the  intervening  years  it  may  be  observed  that  student  performance  in 
mathematics  has  suddenly  dropped.  That  would  merit  a  review.  Any  such 
evaluations  would  be  made  on  the  basis  of  need  rather  than  traditional  scheduling 
of  program  or  school  evaluations. 
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Concluding  Statement 

The  study  has  resulted  in  the  development  of  a  comprehensive  school  system 
review  that  examines  the  full  operation  of  a  school  system  in  one  process. 
Accountability  and  general  public  support  for  education  have  the  potential  to 
improve  as  a  result  of  reporting  the  comprehensive  findings  and  improvement 
plans.  The  interest  of  parents  in  the  school  systems  has  been  heightened  because  of 
their  involvement  in  the  project  and  growth  in  their  knowledge  and  contribution  to 
education  has  occurred.  With  its  focus  on  student  learning  and  related  conditions 
and  processes,  the  review  is  a  powerful  tool  to  describe  the  state  of  student 
learning,  to  identify  strengths  and  weaknesses,  and  through  planning  and  the 
luxury  of  time,  to  make  improvements  in  learning. 
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District  and  School  Profiles 
for  Quality  Education 


Grande  Prairie  School  District  No.  2357 


The  Grande  Prairie  Educational  Quality  Indicators  Project  developed  prototype 
annual  school  and  district  profiles  to  provide  interested  stakeholders  with  current 
and  valuable  information  to  judge  the  health  of  their  educational  system  and  to 
identify  areas  needing  attention.  Annual  feedback  allows  schools  to  become  more 
efficient  and  effective  in  delivering  education  to  their  clients.  The  report  outlines 
the  preliminary  work  in  involving  educational  stakeholders  to  identify  indicators 
of  educational  quality  within  the  school  district  and  describes  methods  to  measure 
and  report  information  in  a  manner  that  is  meaningful  and  useful  to  stakeholders. 
The  benefits  of  gathering  and  using  indicators  to  assess  educational  quality  and  to 
plan  educational  improvements  is  also  discussed. 


Rationale 

The  Grande  Prairie  Public  School  District  became  involved  with  the  three-year 
Educational  Quality  Indicators  initiative  specifically  to  meet  four  educational  goals. 
First,  the  district  was  interested  in  developing  a  set  of  indicators  of  educational 
quality  that  reflect  the  views  of  the  local  educational  stakeholders.  Second,  the 
district  wanted  to  report  the  comparative  results  of  the  local  indicators  in  an  annual 
document  which  would  reflect  the  quality  of  education  within  our  district.  The  third 
goal  was  the  development  of  a  process  to  use  annual  reports  to  plan  educational 
improvement  within  the  district  The  final  goal  was  to  test  the  indicators  and 
processes  in  another  school  district  to  test  their  transferability. 


Purpose 

The  Educational  Quality  Indicators  Project  was  designed  to  identify,  measure  and 
report  on  indicators  of  quality  education  as  defined  by  our  educational  stakeholders 
in  the  Grande  Prairie  Public  School  District.  During  the  three  years  of  the  project, 
contact  was  made  with  approximately  4,000  educational  stakeholders  in  the  city. 
The  contact  made  with  the  stakeholders  provided  the  project  team  with  the 
indicators  used  and  also  the  level  of  stakeholder  satisfaction  with  these  indicators. 
The  project  developed  measures  for  all  the  identified  indicators  that  span  the 
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cognitive,  affective  and  behavioral  domains.  The  annual  reports  on  the  indicators 
and  their  measures  will  provide  stakeholders  with  historical  data  on  which  to  assess 
and  improve  education  in  the  city  schools. 

Two  assumptions  were  made  at  the  onset  of  the  project.  The  first  was  that 
educational  stakeholders  wanted  and  needed  data  which  they  could  understand  and 
which  they  agreed  reflected  the  educational  health  of  the  schools  and  the  school 
district.  During  the  interview  process  in  the  first  year  of  the  study,  the  project  team 
found  this  assumption  to  be  correct.  All  stakeholder  groups  expressed  confusion 
and  frustration  with  the  use  of  standardized  tests  as  the  main  indicator  of  student 
success  in  the  schools.  They  expressed  a  strong  desire  to  use  other  measures 
which  more  clearly  reflected  the  success  of  the  schools  and  their  students. 

The  second  assumption  was  that  the  existing  stakeholders  did  not  have  a  clear 
understanding  of  the  present  educational  system  and  how  it  performed.  This 
assumption  also  proved  to  be  somewhat  true  when  the  team  interviewed 
stakeholders  who  did  not  have  direct  contact  with  the  system.  The  general  public 
and  parents  seemed  to  base  many  of  their  views  on  the  system  they  encountered 
when  they  were  students  in  school.  It  was  also  apparent  to  the  project  team  that 
employees  and  students  did  not  have  an  understanding  of  the  educational  system 
or  product  beyond  their  own  school  building. 


Design 

The  Grande  Prairie  Project  was  designed  in  three  phases,  one  for  each  of  its 
three  years.  During  the  first  phase,  the  project  team  conducted  interviews  with 
representative  groups  to  establish  indicators  of  quality  education  which  would 
serve  as  the  basis  for  the  annual  profiles.  The  interviews  were  conducted  using  the 
nominal  group  technique  and  resulted  in  the  team  making  contact  with  more  than 
1,900  stakeholders.  The  team  also  conducted  an  extensive  review  of  the  literature 
related  to  effective  schools  and  quality  education.  The  final  objective  of  the  first 
phase  was  to  have  a  workable  set  of  definitions  and  indicators  to  continue  into  the 
second  phase  of  the  project. 

Phase  two  was  dedicated  to  developing  workable  criteria  for  measuring  the 
indicators  identified  during  phase  one.  The  initial  task  for  the  team  was  to  cluster 
the  indicators  under  larger  indicator  titles.  The  final  result  of  the  clustering  process 
was  the  establishment  of  four  areas:  student  achievement,  school  climate,  quality 
of  instruction,  and  funding.  The  second  task  for  the  team  was  to  develop  a  measure 
for  each  indicator  and  to  explore  different  methods  to  collect  information  on  the 
indicators.  The  stakeholder  satisfaction  survey  was  also  developed  during  phase 
two  as  an  instrument  to  contribute  to  many  of  the  indicators  identified. 

The  final  phase  of  the  project  was  designed  specifically  to  develop  a  method  of 
reporting  to  stakeholders  on  the  identified  indicators.  This  phase  also  resulted  in 
work  with  the  district  and  the  schools  to  develop  effective  methods  to  gather  and 
report  on  the  data  required.  The  result  of  the  final  phase  saw  the  production  of 
school  and  district  profiles  in  our  pilot  schools  and  in  the  district. 
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School  and  District  Profiles 


Table  1  presents  the  quality  indicators,  how  they  are  measured,  and  their  sources. 
Student  achievement  includes  performance  on  provincial  achievement  tests  and 
diploma  exams,  success  of  graduates,  fitness,  teacher  marks,  affective  education, 
retention  and  attendance.  School  climate  includes  student  and  staff  morale, 
turnover,  behavior,  and  dropouts.  Quality  of  instruction  includes  professional 
attributes,  teacher  inservice,  instructional  methods  and  expectations.  Funding 
includes  cost  efficiency  and  staffing. 

The  project  produced  three  major  products.  The  first  was  the  stakeholder 
satisfaction  surveys.  These  questionnaires  were  designed  to  obtain  feedback  on  the 
four  main  indicator  groups.  Four  types  of  stakeholders  were  surveyed  -  parents/ 
guardians,  students,  district  staff,  and  the  public.  The  questionnaires  will  be 
evaluated  at  the  end  of  each  year  to  determine  what  changes  need  to  be  made  for  the 
next  survey.  The  surveys  were  delivered  to  the  schools  where  participants  were 
asked  to  record  their  answers  on  a  scantron  computer  score  sheet.  This  method  of 
collecting  and  recording  the  stakeholder  responses  was  determined  by  the  team  to 
be  very  successful.  Two  groups  of  stakeholders  did  not  fill  out  the  computer  score 
sheet  directly.  The  general  public  was  contacted  by  trained  student  operators  and 
their  responses  recorded  on  the  computer  sheets.  The  parents  of  the  Composite 
High  School  were  also  contacted  by  a  trained  survey  operator.  Students  of  the 
other  pilot  schools  were  asked  to  deliver  their  parent  surveys  directly  and  all  survey 
sheets  returned  to  the  school  in  good  condition  allowed  the  student's  name  to  be 
entered  into  a  cash  draw.  This  format  for  returns  was  very  successful  at  the  junior 
high  school  level.  Another  product  from  the  surveys  was  the  adaptation  of  existing 
computer  software  packages  to  produce  the  graphed  results  of  each  question  in  the 
survey. 

Table  2  presents  the  stakeholder  responses  to  the  May  1991  survey.  Annual  survey 
results  are  an  integral  part  of  school  improvement  planning.  Survey  results  flag 
areas  that  may  become  the  focus  for  annual  school  or  district  improvement  plans. 
Once  an  area  is  flagged  and  investigated  using  other  indicators,  it  may  become  the 
target  for  improvement.  The  surveys  also  provide  feedback  on  the  results  of  the 
improvement  action  plan. 

The  second  major  product  was  the  individual  school  profile.  These  profiles  serve 
as  the  communication  link  between  the  stakeholders  and  the  school  on  the  quality 
of  education  within  the  school  and  the  actions  schools  are  taking  to  maintain  or 
improve  that  quality.  The  school  profiles  include  reports  on  indicators  requested  by 
the  school  district  as  well  as  indicators  judged  valuable  by  site  teams.  A  subsequent 
product  from  the  school  profile  will  be  the  development  of  a  software  package  mat 
will  allow  schools  to  produce  profiles  with  minimum  labor  and  low  cost. 

The  third  product  was  the  comprehensive  school  and  district  profile.  This  profile 
reported  all  the  indicators  identified  by  the  stakeholders  to  reflect  a  quality 
education.  Each  indicator  has  a  method  of  collecting  and  reporting  representative 
data.  A  major  software  package  is  being  developed  to  allow  schools  and  the  school 
district  to  report  on  all  indicators  in  an  efficient  and  effective  manner. 
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Table  1 
Quality  Indicators 


Indicator 


Measures 


Sources 


Student  Achievement 

provincial  achievement  tests 
diploma  exams 
post  graduate  success 
Canada  Fitness  Award 
teacher  assigned  marks 
affective  education 
retention  rates 
attendance 


grades  3,  6  &  9  students 
grade  12  students 
graduate  questionnaire 
medals  count 
marks  table 

programs,  awards,  participation 
district  statistics 
district  statistics 


School  Climate 

climate 

student  morale 

staff  morale 

turnover 

behavior 

attendance 

lates 

expulsions 
dropouts 

Quality  of  Instruction 

professional  attributes 

inservice 

substitute  days 

instructional  methods 

expectations 

monitoring 

curriculum 


questionnaire 
questionnaire 
questionnaire 
district  statistics 
district  statistics 
district  statistics 
district  statistics 
district  statistics 
district  statistics 


personnel  records 
budget  expenditures 
school  reports 
site  administrator 
school/district  policy 
site  administrator 
administrator 


Funding 


cost  efficiency 
staffing 


secretary-treasurer 
personnel  officer 
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Table  2 

Grande  Prairie  Stakeholder  Satisfaction  in  May  1991  (Percent) 


Indicator 

Parents 

Elementary 

Secondary 

Matt 

n,,i.i:  _ 

Public 

Mudents 

Mudents 

(N=456) 

/"\r_     1  \ 

ViN  —  D  /  L) 

ViN  —  1  /  0; 

Student  Achievement 

Satisfied 

66 

76 

54 

75 

55 

Dissatisfied 

12 

1  A 

14 

ly 

1  o 
1Z 

1  Q 

lo 

Need  Information 

14 

y 

1  /i 
14 

o 
O 

ZO 

Unimportant 

0 

r\ 
U 

U 

rv 
U 

1 

No  Response 

8 

Z 

0 

1 
1 

School  Climate 

Satisfied 

73 

74 

59 

62 

57 

Dissatisfied 

10 

1  A 

14 

23 

25 

18 

Need  Information 

9 

9 

12 

6 

21 

Unimportant 

0 

U 

U 

1 

Z 

No  Response 

8 

a 

D 

O 

0 

i 
1 

Quality  of  Instruction 

Satisfied 

67 

75 

62 

62 

53 

Dissatisfied 

9 

1 i 

11 

Zo 

lj 

Need  Information 

16 

1  f\ 
1U 

1  1 

c 

J 

zy 

Unimportant 

0 

r\ 
U 

0 

1 
1 

1 

1 

No  Response 

8 

z 

U 

0 

z 

Funding 

Satisfied 

69 

65 

69 

48 

76 

Dissatisfied 

8 

22 

24 

35 

8 

Need  Information 

12 

10 

7 

8 

13 

Unimportant 

1 

0 

0 

2 

3 

No  Response 

10 

3 

0 

7 

0 

Note.  Percentages  may  not  total  100  due  to  rounding. 
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The  products  developed  are  invaluable  when  used  to  accomplish  the  intent  of  the 
project  -  to  provide  stakeholders  with  quality  information  on  which  to  assess  and 
improve  the  quality  of  education  in  the  school  district.  The  ultimate  product  of  the 
project  is  the  action  plan.  The  plan  for  using  the  products  of  the  project  is  a  four- 
step  process.  The  first  step  involves  the  formation  of  site-based  improvement 
teams  composed  of  stakeholders  who  assess  the  information  available  to  them  in  the 
comprehensive  school  profiles.  This  site-based  team  then  targets  areas  to  improve 
for  the  upcoming  school  year.  Once  the  areas  of  interest  have  been  targeted,  the 
second  step  of  the  process  is  initiated.  This  step  involves  the  formation  of  an 
action  team  to  develop  a  strategy  that  involves  an  action  plan,  timelines  and  an 
evaluation  method  to  address  the  area  of  concern.  Once  the  action  plan  has  been 
developed,  the  third  step  of  the  process  is  to  formally  implement  the  action  plan. 
The  fourth  and  final  step  is  to  assess  the  results  of  the  improvement  action  based 
on  the  indicators  developed  from  the  project.  The  final  assessment  may  result  in 
abandonment  of  the  action  plan  if  the  improvement  measure  is  negligible  or 
inefficient  due  to  labor  requirements  to  affect  acceptable  change,  or  altering  the 
existing  action  plan  to  provide  more  effective  results  or,  finally,  celebrating  the 
success  of  the  action  plan. 

The  format  for  using  the  products  will  provide  each  school  and  the  school  district 
with  the  ability  to  address  areas  of  need  specific  to  their  own  environment.  It  is 
expected,  however,  the  school  district  will  require  all  schools  to  report  on  common 
indicators  of  educational  quality. 


Conclusions 

The  implementation  of  an  action  plan  involved  the  Grande  Prairie  Public  School 
District  and  the  County  of  Grande  Prairie.  Each  district  provided  volunteer  pilot 
schools  to  begin  the  development  of  school  profiles.  Once  identified,  our  project 
team  members  informed  the  school  administrators  of  the  collection,  presentation 
and  process  involved  in  the  development  of  school  profiles.  The  comprehensive 
school  and  district  profiles  contain  the  results  of  all  annual  data  collection  and  are 
accessible  to  all  stakeholders  in  the  school  district  upon  request.  The  format  of  the 
district  report  does  not  specifically  identify  individual  schools  but  blend  the  results 
into  a  district  profile.  Public  profiles  will  be  condensed  versions  of  the 
comprehensive  school  and  district  profiles.  The  public  profiles  will  use  short 
descriptions,  charts  and  graphs  to  communicate  on  various  quality  indicators. 

Individual  school  profiles  report  on  specific  areas  as  directed  by  the  superintendent 
and  also  reflect  the  needs  of  the  individual  school  community  resulting  in  different 
profiles  for  individual  schools.  The  quintessential  objective  of  the  school  profile  is 
to  provide  school  stakeholders  with  information  that  is  meaningful  and  useful  to 
them  in  planning  school  improvement  and  measuring  the  educational  health  of  their 
schools.  This  objective  is  the  driving  force  that  necessitates  collaboration  among 
stakeholders  to  monitor  improvement  projects  and  to  initiate  required  changes. 
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Implications 

The  Grande  Prairie  Quality  EQI  Project  initiated  a  process  through  which  the  health 
and  direction  of  education  became  the  communication  of  the  performance  of 
education  in  the  district  to  stakeholders.  The  satisfaction  survey  results  emphasized 
the  need  of  the  public  and  parents  for  more  information  on  many  of  the  identified 
indicators.  The  collection  of  the  data  will  become  much  more  efficient  within  the 
schools  and  more  centralized  at  the  district  office.  The  use  of  computer  technology 
to  collect  and  report  on  the  data  will  provide  all  stakeholders  with  information  that 
can  be  easily  accessed  and  properly  interpreted  and  used  in  assessment  and 
improvement  of  the  district.  This  electronic  data  gathering  and  storing  method 
has  put  the  Grande  Prairie  School  District  in  an  excellent  position  to  participate 
in  the  electronic  information  exchange  advocated  by  Alberta  Education. 


Recommendations 

The  sample  of  the  general  public  in  the  project  did  not  include  a  wide  variety  of 
business  and  industrial  representatives.  Future  expansion  or  replication  of  the 
study  should  include  representatives  of  post-secondary  educational  institutions 
and  the  general  public.  The  satisfaction  survey  worked  well  for  the  initial  gathering 
of  information  on  the  identified  indicators.  The  project  team  recommends  the 
development  of  a  validation  process  of  each  of  the  questions  on  the  questionnaires 
over  time  to  further  enhance  the  instruments.  Finally,  a  strong  recommendation  is 
given  to  the  ongoing  annual  collection  and  publishing  of  educational  indicators  for 
the  purpose  of  improving  the  quality  of  education  in  the  school  district. 


Follow-up  -  Ongoing  Project 

The  EQI  Project  is  simply  the  start  of  a  major  effort  within  our  district.  Much  work 
is  ahead  for  stakeholders  to  use  the  indicators  developed  during  the  study  to  plan 
and  monitor  educational  quality.  As  in  any  major  educational  thrust,  stakeholders 
must  be  reinforced  and  encouraged  to  persevere.  It  is  incumbent  on  the 
adniinistration  and  the  board  of  the  school  district  to  publicly  maintain  support 
for  the  direction  of  this  project  toward  quality  education. 

The  Grande  Prairie  Public  School  District  is  committed  to  the  development  of 
profiles  for  ongoing  use  in  the  district.  The  profiles  will  be  adjusted  to  reflect 
the  thirteen  priority  directions  outlined  by  Alberta  Education  in  Achieving  the 
vision  1991  report.  The  Board  of  Trustees  of  the  Grande  Prairie  Public  District 
unanimously  passed  a  motion  at  the  December  8, 1992  meeting  which  directed  each 
school  to  produce  annual  profiles  commencing  in  the  1992  -1993  school  year. 
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Concluding  Statement 


The  initial  efforts  required  to  develop  an  indicator  collection  and  reporting  process 
within  a  school  district  seem  overwhelming.  The  process  must  begin  by  developing 
a  breakdown  of  the  steps  required  to  reach  the  goal  of  the  project.  Once  these  steps 
are  separated,  action  plans  can  be  developed  to  gain  numerous  small  victories  in  the 
progress  toward  the  overall  objective  of  developing  school  improvements  through 
the  collection  and  reporting  of  educational  indicators. 

The  production  of  annual  school  and  district  profiles  will  be  a  continuous  program 
in  the  Grande  Prairie  Public  School  District.  The  instruments  will  provide  interested 
stakeholders  with  current  and  valuable  data  to  not  only  judge  the  health  of  their 
educational  system,  but  also  to  identify  specific  areas  in  need  of  attention.  Annual 
feedback  allows  the  schools  to  measure  progress  or  the  effect  of  efforts  to  improve 
areas  of  education.  If  the  feedback  indicates  the  efforts  have  not  produced  the 
expected  or  desired  results,  school  teams  may  decide  to  abandon  or  change  their 
action  plans.  Feedback  enables  schools  and  school  districts  to  become  more 
efficient  and  effective  in  delivering  education  to  their  clients.  A  more  effective  and 
efficient  educational  system,  as  identified  by  stakeholders,  will  result  in  a  positive 
educational  experience  for  all. 
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A  Collaborative  Model  for  School 
and  Program  Evaluation 


Lethbridge  School  District  No.  51 


In  recent  years  there  has  been  a  marked  increase  in  the  emphasis  and  expectations 
for  school  and  program  evaluations.  Throughout  much  of  Alberta,  the  approach 
used  has  been  predominantly  a  "top  down"  process  with  the  planning  and 
procedures  carried  out  almost  exclusively  by  personnel  from  Alberta  Education 
or  district  central  offices  with  assistance  by  external  resource  persons.  A  meta- 
evaluation,  conducted  by  Lethbridge  School  District  No.  51,  concluded  that  limited 
changes  were  being  realized  by  the  evaluations.  Teachers  and  school-based 
administrators  were  not  actively  involved  and  hence  gained  little  from  the  exercise. 
A  study  of  the  related  literature  and  procedures  used  in  other  locations  led  to  the 
development  of  a  "collaborative  model"  to  empower  school-based  personnel  to  be 
actively  and  professionally  involved  in  their  school  and  program  evaluations.  The 
use  of  "educational  quality  indicators"  served  as  a  key  component  of  the  model. 


Rationale 

The  evaluation  of  teaching  and  of  teachers  has  received  in-depth  study  but  little 
effort  seems  to  have  been  put  into  whether  the  models  and  procedures  used  for  the 
relatively  new  focus  on  school  and  program  evaluations  are  consistent  with  the 
findings  of  educational  research.  Because  the  stated  goal  of  evaluation  is  almost 
always  school  improvement,  there  is  a  need  to  consider  and  utilize  the  research  on 
quality  indicators,  school  effectiveness,  and  school  improvement  A  collaborative 
model  would  ensure  a  "buy  in"  by  those  involved.  This  is  essential  as  evaluation 
alone  does  not  cause  improvement 

In  the  province  of  Alberta,  as  in  other  Canadian  provinces  and  a  number  of 
American  states,  there  has  been  an  increasing  emphasis  and  expectation  from 
governments  that  school  jurisdictions  will  develop  and  carry  out  formalized 
procedures  for  evaluation,  including  the  evaluation  of  students,  teachers, 
programs,  schools,  and  school  systems. 

The  Lethbridge  model  involved  the  use  of  a  large  team  (up  to  55  persons)  from 
the  school  district  central  office,  other  schools,  Alberta  Education  offices,  and  the 
University  of  Lethbridge,  spending  up  to  three  weeks  of  time  on  site  for  the  school 
evaluation  process.  A  common  concern  was  whether  the  model  (which  was 
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extremely  expensive  in  terms  of  professional  time  and  expertise)  was  producing 
payoffs  in  professional  growth  and  development.  Perhaps  the  most  significant 
concern  was  with  the  problem  of  implementing  the  recommendations  emanating 
from  the  evaluation  report. 

Administrators  and  boards  need  assurance  that  their  models  of  school  and  program 
evaluation  are  consistent  with  increased  effectiveness  and  school  improvement  as 
a  result  of  the  investments  of  professional  time,  expertise,  and  budget  dollars. 
Stufflebeam  and  Shinkfield  (1985)  affirmed  that  if  evaluations  are  to  be  useful  and 
provide  proper  direction  and  guidance,  "the  evaluations  themselves  must  be  sound" 
(p.  183). 


Purpose 

The  purpose  of  the  Lethbridge  School  District  No.  51  project  was  to  develop  a 
more  effective  model  for  school  and  program  evaluation.  Consequently,  the 
following  research  questions  were  addressed. 

1 .  Is  the  current  Lethbridge  School  District  No.  5 1  model  for  school  and  program 
evaluation  consistent  with  the  literature  on  indicators  of  effectiveness,  quality, 
and  school  improvement? 

2.  Are  the  procedures,  instruments,  and  data  used  in  the  Lethbridge  School  District 
No.  51  model  valid  and  reliable? 

3 .  Does  the  current  literature  suggest  characteristics  or  criteria  of  more  effective 
and  efficient  models  for  school  and  program  evaluation,  in  terms  of  indicators 
of  effectiveness,  quality,  and  improvement? 

4.  If  the  findings  for  the  above  indicate  "yes",  can  these  findings  be  applied  to  the 
development  of  a  collaborative  model  for  a  school  system  evaluation  of  the  art 
and  library  programs? 


Design 

Warnica  (1989)  completed  a  case  study  of  the  Lethbridge  model  for  school 
and  program  evaluation.  It  was  primarily  qualitative  in  nature,  but  included 
some  correlations.  The  major  areas  of  educational  literature  reviewed  as  a 
basis  for  this  study  and  the  proposed  model  were:  school  and  program 
evaluation  -  theory  and  practice;  effective  schools  research;  indicators 
of  effectiveness  or  quality;  and,  evaluation  models,  criteria,  and  standards. 

After  his  review  of  the  literature  and  based  on  his  career  experience,  Warnica 
(1990)  concluded  that  the  decision  on  the  debate  related  to  the  external  versus 
internal  evaluation  model  falls  clearly  on  the  side  of  the  school-based  model.  The 
convincing  arguments  of  Boud  and  Donovan  (1982),  Common  (1987),  Herman 
(1986),  Toffler  (1980),  Eisner  (1985)  and  others  provide  sound  reasons  for  a 
model  which  involves  more  active  and  professional  participation  of  school-based 
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personnel.  The  empowerment  of  school-based  teachers  and  administrators,  and  the 
school  as  the  focus  of  action  and  development  offer  the  greatest  potential  for  real 
growth  and  development  of  the  school,  its  staff,  and  its  programs  for  students. 

Although  the  intent  of  the  original  research  was  to  focus  on  school  evaluation, 
senior  adrninistrators  and  trustees  directed  the  move  toward  program  evaluation. 
Using  the  study  as  a  basis  of  information,  work  then  proceeded  on  the  application 
of  this  research  to  the  development  of  a  collaborative  model  for  art  and  library 
program  evaluations.  These  programs  were  selected  for  a  number  of  reasons: 

•  a  system  perceived  need  to  evaluate  the  effectiveness  of  these  programs; 

•  availability  of  central  office  leadership;  and 

•  concerns  about  the  lack  of  expertise  of  the  generalist  teacher  in  the  area  of  art 
evaluation  as  well  as  the  lack  of  available  research  on  this  topic. 

Acting  upon  recommendations  from  the  administration,  the  Board  approved  the 
development  and  implementation  of  a  new  evaluation  format.  Steering  committees, 
consisting  of  generalist  and  specialist  teachers  representing  elementary  and 
secondary  schools,  administration,  and  Alberta  Education  personnel,  were 
organized  for  each  program  area.  The  committees  were  given  instruction  and 
direction  on  the  development  of  quality  indicators.  Both  committees  proceeded 
through  the  steps  outlined  in  the  art  program  flow  chart  (Figure  1). 

During  the  second  year  of  the  project,  a  model  was  developed  and  field-tested 
for  evaluating  the  two  identified  programs.  The  collaborative  model  provided 
the  structure  for  application  of  the  effective  schools  research  about  evaluation  and 
the  use  of  indicators  of  effectiveness  or  program  quality.  Key  components  in  the 
development  of  the  educational  quality  indicators  included  the  district-based  staff, 
the  community  and,  the  resulting  determination  of  comparators,  standards,  and 
quality  outcomes  for  all  programs. 

Following  the  collection  of  data  through  interviews  and  the  administration  of 
questionnaires  to  the  various  stakeholder  groups,  analyses  were  conducted. 
As  a  result  of  the  findings,  a  system-  and  school-based  action  plan  was  developed 
and  initiated. 

As  a  result  of  staffing  changes,  identified  system  needs  and  the  availability  of  time, 
the  art  program  became  the  focus  of  the  project  in  the  third  year.  Work  continued 
through  the  various  steps  outlined:  the  identification  and  verification  of  the  student 
outcomes;  and,  the  identification,  development,  and  refinement  of  evaluation  tools 
and  processes.  System  and  teacher  inputs  were  reexarnined  and  the  instruments 
modified  as  required. 


Findings 

The  findings  of  the  case  study  indicated  that  a  new  model  for  school  and  program 
evaluation  should  be  developed.  The  result  of  the  research  was  the  development 
of  a  collaborative  model  which  places  much  greater  responsibility  and  control  in 
the  hands  of  the  stakeholders  -  school-based  administrators,  staff,  students,  and 
parents.  They  became  major  participants  and  decision  makers  in  school  and 


Model  for  Evaluation 


45 


Lethbridge  Public 


Develop 
Collaborative 
Model 


Identify 
Subject  Area 
Evaluation 


Identify  Quality 
Indicators  and 
Outcomes 


Internal 
Self  Evaluation 


 T 

School  | 
Administrators  | 


Identify 
Committee 
Structure 
 W 


Steering 
Committee 


Evaluation 
Team 


Evaluation 
Team  Roles 
Outlined 


10 


Review/Revise 
Quality 
Indicators 


Develop 
Questionnaires 


Evaluation  Team 

Reviews 
Questionnaires 


Administrators 

Approve  of 
Questionnaires 


V  25 


5  24 


Verification  and 
Summarization 
Process 


26 


27 


V 


Review/Revise 
Collaborative 
Eval.  Model 


Develop  .and 
Implement 

Tests 
a  Primary 
b.  Upper  Bern, 
c.  Secondary 


Summarize  and 
Compare 

a.  Mid-evaluation 
Observations 

b.  Self-evaluation 

Results 
c  Test  Results 


28 


Review 
System/Schools 
Action  Plan 


29 


Write  Final 
Report  for 
EQI 


30 


Implement 
Action 
Plan 


"V  31 


EQI 
Conference 


Figure  1:  Art  Program  Evaluation  Flowchart 
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program  evaluations,  in  identifying  areas  of  program  and  professional  growth, 
and  in  bringing  about  change.  The  model  rests  upon  certain  basic  assumptions 
grounded  in  the  literature  on  school  effectiveness  and  school  improvement.  The 
school  is  the  primary  unit  of  decision  making  (Smith  &  Purkey,  1985).  If  changes 
are  to  occur,  they  require  ownership  that  comes  from  the  opportunity  to  participate 
in  defining  change,  and  the  flexibility  to  adapt  it  to  individual  circumstances. 
Change  does  not  come  from  externally  imposed  procedures  (Fullan,  1982). 

Schools  need  to  identify  areas  requiring  improvement  and  work  actively  toward 
this  end.  Empowerment  of  school-based  staff  and  administration  is  fundamental 
to  reviewing  and  improving  schools.  As  a  perception  check,  a  follow-up 
questionnaire  was  distributed  to  the  stakeholders  at  the  conclusion  of  the  art 
evaluation.  The  results  from  this  questionnaire,  in  conjunction  with  a  review 
of  the  content  and  the  initiation  of  the  program  action  plans  developed  by  the 
schools,  confirmed  that  the  collaborative  model  was  preferred. 

The  collaborative  nature  of  this  approach  resulted  in  a  "buy  in"  by  stakeholder 
groups.  The  development  and  use  of  quality  indicators  was  positively  received 
and  their  use  incorporated  into  assessment  procedures. 

As  predicted  by  the  art  steering  committee,  evaluation  procedures  were  identified 
as  an  area  of  concern.  A  third  of  the  over  3,000  students  surveyed  did  not 
understand  how  their  report  card  mark  was  established;  only  7%  indicated  a  clear 
understanding  of  the  process.  Most  parents  (80%)  were  not  aware  of  how  their 
child  was  being  evaluated  and  37%  of  the  teachers  questioned  indicated  problems 
with  evaluation  procedures.  A  follow-up  questionnaire  addressed  to  teachers  and 
administrators  confirmed  that  through  the  application  of  the  collaborative  model 
and  the  identification  of  quality  indicators,  a  positive  change  in  attitude  regarding 
assessment  in  art  had  taken  place. 

The  model  presented  in  Figure  2,  although  originally  developed  for  an  art  program, 
is  basically  a  generic  model  that  can  be  applied  to  other  subject  areas  with  few 
modifications.  Assessment  procedures  to  measure  the  desired  student  outcomes 
can  be  identified  and/or  developed. 


The  Art  Program  Model 

Figure  3  presents  the  Results-based  Art  Program  Plan,  which  is  a  refinement  of 
Figure  2.  It  does  not  include  the  inputs  of  the  home,  community,  and  society, 
some  of  which  are  inherent  under  other  headings.  This  model  encourages  the 
teacher  to  identify  his/her  individual  mission  statement/belief.  It  stresses  the 
importance  of  inputs  into  the  program;  the  need  to  take  students  from  where  they 
are  with  the  preassessment  and  to  track  their  growth  along  the  continuum.  The 
plan  assists  teachers  in  focusing  on  the  desired  student  outcomes  and,  through  the 
use  of  a  variety  of  assessment  procedures,  determine  the  actual  student  outcomes. 
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Implications 

1 .  Based  on  the  current  literature  and  the  practical  application,  it  has  been 
demonstrated  that  a  more  effective  model  for  school  and  program  evaluation 
involving  collaboration  and  the  identification  of  educational  quality  indicators 
can  be  designed  and  implemented. 

2 .  The  development  of  a  clear  set  of  standards  or  indicators  of  educational  quality 
would  be  a  desirable  step  in  moving  the  evaluation  model  on  to  more  objective 
ground,  and  would  allow  much  more  professional  involvement  of  school  level 
personnel. 

3 .  Although  the  art  program  evaluation  was  a  practical  application  of  the  research, 
the  model  developed  could  be  applied,  with  few  modifications,  to  other  subject 
areas.  However,  assessment  procedures  would  have  to  be  identified/developed 
for  the  different  programs. 

4.  Community/stakeholder  involvement  in  the  evaluation  process  has  proven  to  be 
very  valuable  input  and  should  continue  to  be  utilized. 


Recommendations 

1 .  School  and/or  program  evaluations  should  be  designed  using  a  collaborative 
model. 

2 .  School  and/or  program  evaluations  should  include  the  development  and  use  of 
educational  quality  indicators.  A  vital  component  is  the  involvement  of 
stakeholders  in  defining  what  constitutes  quality. 

3 .  School  and  program  evaluation  should  be  an  ongoing  process  and  not  simply 
an  event  which  takes  place  once  every  five  years  or  so.  An  evaluation  model 
should  ensure  that  program  and  school  evaluation  becomes  a  natural  and  vital 
part  of  the  curriculum  and  instruction  cycle. 

4.  The  program  evaluation  model  that  has  been  developed  should  be  applied  to 
other  program  areas  in  Lethbridge  School  District  No.  51  and  is  recommended 
for  other  jurisdictions. 

5 .  The  results  of  this  project  should  be  disseminated  to  other  jurisdictions  by 
Alberta  Education. 

6 .  Further  development  and  validation  of  the  art  program  evaluation  model  should 
continue,  particularly  with  regard  to  the  refinement  of  the  educational  quality 
indicators  and  the  verification  of  student  outcomes  through  assessment 
procedures. 

7 .  Teacher  training  in  the  use  of  a  variety  of  assessment  procedures  that  can  be 
employed  in  measuring  whether  students  have  achieved  the  desired  outcomes  is 
essential. 
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Follow-up 


Lethbridge  School  District  No.  51  is  continuing  to  expand  the  use  of  this  model  to 
other  program  areas.  Revisions/refinements  to  the  collaborative  model  will  be  an 
ongoing  process.  The  identification  and  development  of  assessment  procedures 
for  the  measurement  of  student  outcomes  will  be  expanded  to  other  programs. 

Interest  has  been  expressed  by  the  Calgary  Public  and  Catholic  EQI  Project 
members  to  work  cooperatively  with  Lethbridge  to  merge  the  two  projects. 
This  possibility  will  be  explored. 


Closing  Statement 

As  a  result  of  the  research  project,  it  has  been  determined  that  the  former  model  of 
program  and  school  evaluation  was  not  consistent  with  the  literature  on  indicators 
of  effectiveness,  quality  and  school  improvement.  New  evaluation  models  should 
move  away  from  the  heavy  emphasis  on  a  "top  down"  approach  to  encourage  and 
empower  school  staff  and  administrators  to  be  actively  involved  and  interacting 
in  assessing  the  effectiveness  of  their  own  programs  and  in  planning  their 
improvement.  A  collaborative  model  for  school/program  evaluations  in  which 
the  use  of  "educational  quality  indicators"  is  a  key  component,  is  more  effective. 
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A  Collaborative  Teacher  Evaluation  Model: 
Teachers  Do  Make  a  Difference 


Spirit  River  School  Division  No.  47 


The  Spirit  River  School  Division  Educational  Quality  Indicators  Project  involved 
the  development  and  validation  of  a  set  of  indicators  to  assess  teacher  effectiveness 
in  a  formative  process.  A  major  component  of  this  project  was  the  process.  The 
extensive  involvement  of  teachers  in  the  formative  evaluation  aspects  of  the  project 
impacted  upon  teacher  effectiveness  in  the  classroom  and  student  outcomes.  We 
intend  to  continue  this  project  for  three  more  years  to  determine  its  potential  for 
export  to  other  school  jurisdictions  within  Alberta  and  elsewhere. 


Introduction 

Rationale 

We  believe  that  teachers  do  make  a  difference,  for  as  Broudy  (1975)  stated:  "The 
teacher  is  still  the  key  to  schooling:  with  good  teaching,  almost  any  curriculum, 
school  organization,  and  administrative  invention  seems  to  succeed"  (p.  64). 
Experience  and  the  literature  clearly  suggest  that  teacher  and  administrator 
involvement  in  the  selection  of  educational  indicators  was  critical  to  the  success  of 
a  project  of  this  magnitude.  For  any  project  involving  key  components  of  action 
research,  it  was  necessary  to  gain  acceptance,  involvement  and  the  positive  initiative 
of  staff.  The  project  focused  on  professional  development  and  staff  inservice.  A 
collegial,  nonthreatening  model  for  more  effective  teaching  and  benefit  to  students 
using  formative  supervision  and  observation  was  developed  and  implemented  over 
three  years.  It  focused  the  attention  of  the  system  on  the  wider  range  of  student 
educational  outcomes,  affective  and  behavioral  as  well  as  cognitive. 


Purpose 

A  wealth  of  information  on  effective  teaching  behaviors,  substantiated  by  practical 
research,  has  emerged  over  the  last  few  decades.  The  Spirit  River  School  Division 
Indicators  Project  involved  identification  and  validation  of  a  set  of  indicators  that 
can  be  used  to  assess  teacher  effectiveness.  The  criteria  by  which  to  measure 
teacher  effectiveness  are  of  central  importance  to  the  concept  of  teacher  evaluation 
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and  effectiveness.  Normed  measures  of  teacher  behaviors  have  been  established, 
and  student  affective  and  cognitive  outcomes  as  they  relate  to  those  behaviors  are 
being  measured. 

The  question  posed  is:  Do  the  26  teaching  behaviors  we  have  identified  as  being 
critical  to  effective  teaching  have  an  impact  on  student  outcomes  in  both  the 
cognitive  and  affective  domains? 


Design 

This  project  concentrated  on  isolating  and  verifying  established  criteria  for  the 
various  policies  on  teacher,  school  and  subject  evaluations  which  were  already  in 
place.  Ultimately,  the  project  involved  developing  and  validating  a  set  of  indicators 
which  could  be  used  to  assess  teacher  effectiveness  in  a  formative  process. 

Voluntary  support  through  acceptance  and  involvement  was  received  from  staff. 
More  than  80%  of  all  staff  responded  to  a  survey  that  asked  them  to  indicate  the 
criteria  each  felt  were  critical  to  improving  teacher  effectiveness.  Eight  experienced 
and  respected  teachers  expressed  a  keen  interest  in  serving  on  the  local  committee 
whose  purpose  was  to  finalize  the  indicators  and  establish  the  process  and  terms  of 
reference  for  the  project  within  the  established  guidelines. 

This  committee  reviewed  pertinent  research  and  literature,  developed  a  set  of  26 
criteria  of  effective  teaching,  and  formulated  key  variables  which  could  impact  the 
effectiveness  of  implementation  in  all  division  schools  on  a  voluntary  basis.  The 
committee  reviewed  the  input  from  staff  and  finalized  the  working  project 
documents.  Table  1  presents  the  26  indicators. 

During  the  first  year,  on  a  voluntary  basis  and  in  a  formative  process,  all  staff  had 
the  opportunity  to  meet  and  discuss  the  previously  developed  indicators  of  effective 
teaching  or  visit  each  other's  classrooms  to  observe  one  or  more  of  these  specific, 
established  areas.  Data  were  collected  on  staff  participation,  specific  objectives 
related  to  each  type  of  indicator,  local  or  out-of-school  contacts,  and  years  of 
teaching  experience.  Information  was  also  gathered  on  geographic  distance,  use 
of  extra  time  provided  in  staffing,  staff  attitudes,  and  extended  professional 
development  activities.  Two  survey  instruments  were  developed  and  validated 
using  Good's  (1963)  validation  criteria  and  parents  and  students  were  subsequently 
surveyed  to  gain  their  perspectives  of  indicators  of  quality  teaching.  Surveys  were 
distributed  to  all  students  in  grades  9, 10,  11  and  12  and  parents  of  students  in 
grades  2, 5,  8  and  11.  The  intent  of  these  representative  surveys  was  to  gain  parent 
and  student  input  into  the  process  and  criteria,  while  obtaining  their  perception  on 
validity  and/or  development  of  new  criteria.  A  teacher  survey  on  satisfaction  and 
professional  development  involvement  was  also  conducted.  The  data  gathered  at 
the  conclusion  of  the  first  year  of  the  project  were  collated  and  presented  on  a 
system  level,  thereby  assuring  the  anonymity  of  participants. 

In  the  second  year  of  the  project,  a  decision  was  made  to  enhance  the  credibility 
of  the  project  as  a  research  endeavor  through  the  establishment  of  a  teacher 
performance  baseline  to  which  comparisons  could  be  made  at  a  later  point  in  time. 
Teaching  behaviors  with  the  lowest  ratings  were  addressed  at  the  system  level 
through  a  variety  of  professional  development  activities. 
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Table  1 

Indicators  of  Quality  Instruction 


1.  Sequentially  Developed  Planning  and  Preparation 

1.1  The  teacher  maintains/develops  short,  unit  and  long  term,  flexible  plans  with  an  evaluation 
process. 

1 .2  The  teacher  adheres  to  Curriculum/Program  of  Studies  documents. 

1 .3  The  teacher  provides  a  safe  and  organized  environment  with  resources  and  equipment  readily 
available. 

1 .4  The  teacher  provides  for  individual  learning  needs  with  consideration  of  student  abilities, 
based  on  diagnostic  information. 

1 .5  The  teacher  provides  a  variety  of  learning  activities. 

1 .6  The  teacher  ensures  that  objectives  are  stated. 

1 .7  The  teacher  provides  for  motivation  of  students. 

1 .8  The  teacher  makes  learning  activities  relevant 

1 .9  The  teacher  plans  for  interdisciplinary  activities. 

2.  Key  Instructional  Strategies 

2. 1  The  teacher  circulates  during  pupil  activities. 

2.2  The  teacher  presents  clear  assignments  which  are  relevant  to  student  experiences. 

2.3  The  teacher  exhibits  classroom  management  skills. 

2.4  The  teacher  provides  an  environment  whereby  active  learning  takes  place  within  a  working 
and  non-threatening  atmosphere. 

2.5  The  teacher  provides  a  physical  classroom  environment  which  is  stimulating,  enriched, 
relevant  and  has  displays  of  student  work. 

2.6  The  teacher  exhibits  a  variety  of  questioning  techniques. 

3.  Communication  Skills 

3 . 1  The  teacher  provides  clear  explanation  of  objectives,  assignments  and  behavioral  expectations. 

3 .2  The  teacher  exhibits  a  variety  of  positive  reinforcement  techniques. 

3 .3  The  teacher  is  an  active  listener. 

3 .4  The  teacher  provides  timely  and  relevant  reviews. 

3.5  The  teacher  uses  language  appropriate  to  the  developmental  level  of  students. 

3 .6  The  teacher  maintains  regular  parental  information  flow. 

4.  Pupil/Teacher  Relationships 

4 . 1  Mutual  respect  is  exhibited  between  teacher  and  students. 

4.2  Clear  routines,  expectations  and  rules  are  made  known  and  followed. 

4.3  The  teacher  provides  for  positive  peer  interaction  and  support. 

4.4  The  teacher  uses  a  fair  and  meaningful  evaluation  process. 

4.5  The  teacher  provides  prompt  feedback  on  student  assignments. 
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A  need  was  also  recognized  for  additional  student  outcome  measures  relevant  to  the 
intent  of  the  indicator  system.  The  Educational  Quality  Indicators:  Inventory  of 
Assessment  Instruments  (Alberta  Education,  1990)  was  consulted.  Sample  copies 
of  a  number  of  assessment  instruments  were  reviewed  and  the  Classroom 
Environment  Scale  (Moos  &  Trickett,  1987)  was  selected  for  grades  5,8,  and  1 1 
and  the  Student's  Perception  of  Ability  Scale  (Boersma  &  Chapman,  1977)  was 
selected  for  grade  2  to  measure  student  outcomes  in  the  affective  domain.  The 
former  served  to  supplement  data  gained  on  cognitive  outcomes  measured  by 
provincial  achievement  tests  and  diploma  exams.  The  data  collected  in  the  third 
year  were  identical  to  those  of  the  second  year  of  the  project.  Table  2  summarizes 
the  data  collected  during  the  second  and  third  years. 


Findings 

Student  outcomes  must  be  monitored  over  an  extended  period  of  time.  Only  then 
can  comparisons  be  made,  trends  or  themes  identified  and  results  substantiated. 
As  a  result,  only  conclusions  relating  to  strategies  that  have  been  successful  in 
promoting  the  active  involvement  of  professional  staff  in  this  change  process  are 
being  shared,  along  with  a  very  brief  reference  to  parental  perception. 


Staff 

The  active,  voluntary  involvement  of  professional  staff  in  action  research  was  the 
most  positive  aspect  of  our  EQI  project.  The  value  teachers  attached  to  this  project 
in  terms  of  the  professional  growth  opportunities  it  provided  was  clearly  illustrated. 
Approximately  90  percent  of  all  teachers  responding  to  the  survey  felt  their 
involvement  contributed  to  their  professional  growth. 

Direct  staff  input  into  the  development  and  validation  of  a  set  of  indicators  that  can 
be  used  to  assess  teacher  effectiveness  in  both  formative  and  summative  processes 
provided  a  sense  of  ownership  and  subsequently  enhanced  the  likelihood  of 
continued  participation.  The  number  of  classroom  visitations  made  each  year  is 
indicative  of  teachers'  willingness  to  be  active  participants  in  this  process. 

Research  provides  considerable  evidence  to  support  the  position  that  teachers 
should  play  a  significant  role  in  setting  the  criteria  by  which  their  performance 
is  to  be  judged.  Teachers  in  this  jurisdiction  did  set  the  criteria  to  be  used  in  the 
formative  evaluation  process,  and  subsequendy  supported  adoption  of  these  same 
criteria  for  the  summative  process.  Ownership  is  having  a  positive  impact  upon 
teachers'  willingness  to  accept  these  as  legitimate  criteria. 

Teachers  have  shown  a  willingness  to  modify  teaching  behaviors.  Table  3 
illustrates  a  change  in  the  modal  rating  on  a  considerable  number  of  criteria. 
Similarly,  the  mean  rating  has  increased  significantly  on  a  number  of  criteria. 
Clear  description  of  the  meaning  of  each  criterion  resulting  from  discussions 
with  colleagues  and  administrators  may  have  contributed  to  this  positive  change. 
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Table  2 
Data  Collection  1990  to  1992 


Type 


Indicator 


Measure 


Source 


Staff 


Parent 


participation 


professional 
development 


teacher 
effectiveness 


perceptions 


number  of  visitations 
made  to  other  teachers' 
classrooms 

number  of  teachers 
attending  out-of-division 
PD  activities 

aggregate  rating  on 
each  of  the  26  teaching 
behaviors 

survey  of  parents  of 
students  in  grades 
2,  5,  8,  11 


teachers 


teachers 


evaluator 


parents 


Student       perceptions        survey  of  students  in  students 

grades  9,  10,  11,  12 

attitudes  Student's  Perception  of  students 

Ability  Scale  (grade  2) 
Classroom  Environment  Scale 
(grades  5,  8,  11) 

achievement       grades  3, 6  and  9  students 
achievement  tests 
grade  12  diploma 
examinations 

discipline  suspensions,  expulsions,  administrators 

referrals  to  office,  dropouts 
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Teachers  actively  participated  in  the  review  and  major  revision  of  our  student  and 
school  evaluation  policies  and  processes  to  bring  them  "on  line"  with  the  indicators 
we  have  adopted.  Two  documents,  a  school  improvement  handbook  and  a  student 
evaluation  handbook  were  the  products  of  this  project,  both  germane  to  promoting 
effective  teaching.  Under  divisional  sponsorship,  a  local  committee  of  teachers 
produced  a  new  teacher  plan  book  which  we  believe  will  have  a  positive  impact  on 
planning  and  reporting  procedures. 

The  teacher  questionnaire  indicated  strong  support  for  the  professional  growth 
opportunities  this  project  has  provided.  Typical  written  responses  follow. 

"A  good  project  that  promotes  and  facilitates  teacher  development." 

"I  think  this  EQI  project  is  very  valuable  to  all  teachers.  It  gives  a  good 
opportunity  to  observe  others  in  action  as  well  as  to  receive  input  to  help 
enhance  our  classroom  environments." 


Parents 

With  few  exceptions  the  parental  survey  results  indicated  a  high  level  of  support  for 
the  criteria  currently  being  used  to  judge  teacher  performance.  A  safe  and 
productive  environment,  attention  to  individual  learning  needs,  clarity  of  objectives, 
the  teacher  being  an  active  listener,  and  the  entire  area  of  pupil-teacher  relationships 
are  the  highest  priorities  of  parents.  These  factors  have  the  highest  impact  on 
student  attitudes  toward  school  and  subsequently  on  academic  performance. 


Students 

In  all  three  years  there  was  generally  reasonable  consistency  in  the  responses  of 
parents  and  students.  Student  opinion  on  any  of  the  criteria  was  not  as  strong  as 
that  of  parents.  Without  exception  a  significant  proportion  of  students  indicated 
"less  important"  as  their  rating  of  any  particular  criterion.  Overall,  however,  the 
only  significant  disagreement  between  parents  and  students  was  regarding  the  flow 
of  information  to  the  home  and  the  importance  of  the  teacher  circulating  in  the 
classroom.  Students  attached  much  less  importance  to  these  than  did  parents. 


Conclusions 

At  this  point  in  time  it  is  premature  to  speculate  on  the  impact  modification  of 
instructional  strategies  is  having  upon  student  outcomes.  As  a  result,  only 
conclusions  relating  to  strategies  that  have  been  successful  in  promoting  the  active 
involvement  of  professional  staff  in  this  process  can  be  shared.  Other  jurisdictions 
may  see  potential  value  for  incorporating  components  of  these  strategies  into  plans 
of  action  they  may  be  pursuing. 
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Table  3 

Teacher  Performance  in  1991  and  1992 


Criteria  1991  1992 

(N=82)  (N=74) 
Mean    Mode       Mean  Mode 


Sequentially  Developed  Planning  and  Preparation 


1.1  plans 

4.3 

4 

4.3 

5 

1.2  curriculum 

4.3 

4 

4.8 

5 

1.3  environment 

4.3 

4 

4.5 

5 

1.4  individual  needs 

4.2 

4 

4.1 

5 

JL.J  vaiiciY 

4  0 

4 

*T. 

1.6  objectives 

4.0 

4 

4.0 

5 

1.7  motivation 

4.0 

4 

4.2 

4 

1.8  relevance 

4.1 

4 

4.3 

5 

1  0  infprHi  viTYHtiarv 

X.7    llit^XvUOV'lLlllllCU  y 

4 

4 

Key  Instructional  Strategies 

2.1  circulation 

3.8 

5 

4.4 

5 

2  2  claritv 

4.1 

4 

4.4 

4 

2.3  management 

4^2 

5 

5 

2.4  atmosphere 

4.3 

5 

4.4 

5 

2.5  environment 

3.9 

4 

4.0 

5 

2.6  questioning 

3.9 

4 

4.0 

4 

Communication  Skills 

3  1  PYnlanatinn 

4  1 

4 

4  1 

3.2  reinforcement 

3.6 

4 

4.1 

4 

3.3  listening 

4.2 

4 

4.3 

5 

3.4  review 

3.9 

4 

4.3 

5 

3.5  language 

4.2 

4 

4.6 

5 

3.6  parents 

4.0 

5 

4.0 

4 

Pupil/Teacher  Relationships 

4.1  respect 

4.4 

5 

4.3 

5 

4.2  expectations 

4.2 

5 

4.2 

5 

4.3  interaction 

4.2 

4 

4.3 

5 

4.4  evaluation 

4.0 

5 

4.3 

4 

4.5  feedback 

4.1 

4 

4.4 

5 

Rating  scale:  1  -  very  poorly  addressed;  2  - 

•  poor;  3 

-  fair;  4  -  good;  5 

-  excellent. 
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With  the  support  and  involvement  of  staff,  we  believe  the  project  is  proving  to  have 
a  very  positive  and  professionally  rewarding  impact  on  teachers  for  the  benefit  of 
students.  It  is  the  impact  on  student  outcomes  that  will  continue  to  be  the  focus  of 
this  study. 


Implications  and  Recommendations 

Based  on  the  findings  of  this  study,  the  following  implications  were  derived. 
Reference  is  made  exclusively  to  teachers  as  they  were  the  "key  players"  in  this 
process. 

Staff  support  is  crucial  to  the  success  of  any  intervention  strategy.  Without  the 
voluntary  support  of  teachers,  a  study  of  this  nature  and  magnitude  would  not  have 
been  possible.  Mandates  were  not  a  part  of  this  process.  Staffs  must  be  given  the 
opportunity  for  voluntary  involvement  in  the  very  early  stages  of  any  project  or 
study  similar  to  this.  This  enhances  the  likelihood  of  acceptance  and  participation 
over  an  extended  period  of  time. 

Teachers  must  have  the  opportunity  and  the  channels  for  "feeding"  into  the  system. 
This  generates  an  allegiance  to  the  system  of  which  they  are  a  part.  Under  the 
leadership  of  central  office  adiriiriistration  all  decisions  relating  to  this  project  were 
made  by  a  local  committee  of  teachers.  This  included  the  identification  of  indicators 
of  effective  teaching,  formulation  of  key  variables  which  could  impact  the 
effectiveness  of  implementation,  identification  of  26  teaching  behaviors  perceived 
critical  to  positive  student  outcomes,  articulation  of  clear  explanations  of  objectives, 
development  and  selection  of  data  gathering  instruments,  and  the  development  of  an 
action  plan  for  implementation  in  all  division  schools. 

Direction  and  support  from  the  adrninistration  and  the  school  board  is  critical  to  the 
potential  for  significant  changes  being  made.  The  school  board's  commitment  to 
this  project,  evident  in  the  funding  it  provided  to  cover  the  cost  of  substitute 
teachers,  the  addition  of  a  staffing  component  to  all  small  schools,  and 
the  priority  given  the  project  in  the  division's  Three  Year  Plan  were  perceived  as 
support  for  the  project  and  its  intent.  This  has  provided  the  time  teachers  needed 
for  planning  and  implementation. 

Tangible  benefits  or  results  enhance  the  likelihood  of  continued  commitment  and 
participation.  Rewards,  whether  intrinsic  or  extrinsic,  serve  as  motivational  tools. 


Project  Follow-up 

The  Board  of  Trustees  of  the  Spirit  River  School  Division,  with  the  support  of 
senior  administration,  has  made  a  commitment  to  carry  on  with  the  study.  Funding 
has  been  budgeted  to  cover  costs  of  substitute  teachers  and  the  additional  staffing 
component  in  our  small  schools  for  the  forthcoming  school  year. 

The  extensive  data  collection  currently  in  place  will  be  continued.  We  are  now 
completing  only  our  first  measurements  of  the  effectiveness  of  modifying 
instructional  strategies.  In  order  that  we  may  appropriately  assess  the  results  and 
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establish  relationships,  should  the  results  support  this,  additional  years  of  research 
and  documentation  are  required.  To  stop  the  process  at  this  point  would  have  a 
negative  impact  on  the  accomplishments  of  the  last  three  years. 

It  is  also  our  intent  to  continue  to  inform  our  publics  of  the  efforts  being  directed  at 
making  our  schools  a  better  place  for  students.  The  local  media  and  our  school 
newsletters  will  continue  to  serve  as  the  primary  disseiriinators  of  information  to  the 
public.  Discussions,  productions  and  presentation  at  special  events  such  as 
Education  Week  will  continue  to  augment  this  process.  Our  ability  to  demonstrate 
to  our  publics  that  education  is  an  excellent  investment  in  our  children  and  our 
future  will  continue  to  be  a  high  priority. 


Concluding  Statement 

Teachers  do  make  a  difference!  This  EQI  project  has  been  a  positive  experience  for 
many  stakeholders  in  the  Spirit  River  School  Division.  We  believe  that  the  project 
is  already  having  a  very  strong  and  professionally  rewarding  impact  on  teachers  for 
the  benefit  of  our  students. 

Using  the  same  criteria  for  both  summative  and  formative  evaluations  has 
heightened  teacher  awareness  regarding  professional  expectations.  The  teacher's 
planbook  has  been  well  accepted  in  its  probationary  year.  It  will  be  revised, 
enhanced  and  updated  to  best  meet  the  needs  of  staff  in  our  division. 
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High  Success  in  Mathematics 


Through  Outcome-Based  Education 


Fort  McMurray  RCSSD  No.  32 


The  purpose  of  the  Outcome-Based  Education  (OBE)  Project  was  to  determine 
whether  OBE  makes  a  positive  difference  in  student  achievement,  responsibility, 
and  attitudes  in  mathematics.  This  report  tells  the  story  of  the  OBE  project.  Teachers, 
students  and  parents  are  the  real  heroes  of  this  story.  The  results  are  a  testimony  to 
their  beliefs,  efforts  and  dedication.  The  findings,  based  on  a  variety  of  indicators, 
show  the  results  of  the  OBE  process.  It  is  evident  that  the  OBE  process  has  made 
a  positive  difference  in  the  outcomes  of  student  achievement,  responsibility  and 
attitudes. 


Introduction 

Outcome-Based  Education  is  a  comprehensive  restructuring  process.  It  is  not  a 
program.  Several  traditional  educational  practices  are  targeted  for  restructuring 
through  the  OBE  process.  The  practices  deal  with  what  we  want  all  students  to 
demonstrate,  how  well  we  want  and  expect  them  to  demonstrate  it,  and  when  and 
how  we  give  them  credit  for  having  demonstrated  success. 

The  restructuring  efforts  in  the  project  were  organized  around  the  four  principles 
of  OBE:  a)  designing  down  from  where  we  want  to  end  up,  b)  a  clear  focus  on 
outcomes  of  significance,  c)  high  expectations  and  standards  for  the  success  of  all 
students  and  d)  time  to  learn  with  appropriate  opportunities  for  learning. 


Rationale 

The  Fort  McMurray  Catholic  Schools  welcomed  an  invitation  from  Alberta 
Education  to  collaboratively  determine  how  well  our  students  were  achieving 
essential  learning  outcomes. 

We  were  interested  in  collecting  data  that  would  allow  us  to  assess  the  effects 
of  an  OBE  restructuring  process.  The  principles  of  OBE  were  well  supported  in 
the  educational  research.  The  OBE  team  wished  to  align  its  practices  with  the 
principles  to  increase  student  success.  Our  hypothesis  was:  If  the  OBE  principles 
are  separately  supported  in  the  research  as  effective  practices,  they  should  produce 
significant  results  in  student  achievement  when  introduced  simultaneously. 
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The  OBE  team  understood  that  the  restructuring  process  necessitated  a 
comprehensive  transformation  of  some  educational  practices,  beginning  with 
our  vision  of  student  success.  Our  focus  on  improvement  and  data  collection 
was  in  one  subject  area  since  we  had  been  cautioned  to  "start  small".  The  reason 
for  our  initial  focus  on  mathematics  was  brought  about  by  the  superintendent's 
understanding  of  results  and  his  vision  for  improved  student  demonstrations  of 
student  achievement  in  mathematics. 


Purposes 

Our  project  purposes  were  to  implement  OBE  in  two  schools  and  to  review  the 
effects  of  our  efforts  on  student  achievement,  responsibility  and  attitudes  in 
mathematics. 


Design 

Data  Sources 

To  collect  the  data  annually,  the  OBE  team  decided  to  use  a  variety  of  measures  and 
data  sources  to  provide  a  comprehensive  picture  of  student  success  in  the  desired 
outcomes.  The  various  measures  and  data  sources  used  in  the  project  are  identified 
in  Table  1. 

Provincial  Achievement  Tests  -  Provincial  mathematics  achievement  tests  were 
written  by  students  in  the  official  writing  years  with  repeated  use  of  the  measure 
in  the  following  years.  The  tests  were  duplicated  with  permission  from  Alberta 
Education.  Teachers  received  the  tests  on  the  morning  of  administration,  supervised 
the  students'  writing  of  the  test,  and  forwarded  the  tests  to  be  scored  by  a  district 
team.  The  purpose  in  using  these  provincial  tests  was  to  determine,  through  reliable 
and  valid  measures,  whether  the  OBE  process  showed  achievement  gains  over  time. 
Although  comparisons  were  made  with  provincial  cohorts,  the  major  comparisons 
we  wanted  to  see  were  the  comparisons  within  a  school  over  time. 

Teacher-Developed  Tests  -  These  tests  were  used  in  grades  2  to  10.  They  provided 
the  data  used  in  reporting  learning  progress  to  students  and  parents.  Other  sources 
of  information  for  progress  reporting  included  samples  of  student  work  in 
portfolios  and  curriculum  aligned  checklists. 

Local  Surveys  -  Local  stakeholder  surveys  were  developed  and  included  questions 
related  to  each  of  the  identified  outcomes.  Survey  statements  were  rated  on  a  5- 
point  scale: 

Not  at  all        A  little  Moderately     Quite  a  lot      Very  much 

(1)  (2)  (3)  "  (4)  (5) 

Statements  to  which  parents  responded  included: 

•  I  am  aware  of  the  High  Success  process  in  my  child's  class. 

•  My  child  has  shown  an  increase  this  year  in  achievement,  responsibility 
and  positive  attitudes 
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Table  1 


Educational  Quality  Indicators/Outcome-Based  Education 
Data  Collection 


Student 
Outcome 

Measures 

Sources 

Grades 

achievement 

Alberta  Education 
mathematics  achievement  tests 

students 

3,  6,9 

teacher-developed  tests 

students 

3  to  10 

local  surveys 

students 
teachers 
parents 
administrators 

3  to  10 
z  to  1U 
2  to  10 
2  to  10 

responsibility 

local  surveys 

students 
teachers 
parents 
administrators 

3  to  10 
2  to  10 
2  to  10 
2  to  10 

random  interviews 

students 
parents 
teachers 
support  staff 
auJTuniSLraiors 

3  to  10 
3  to  10 
2  to  10 
2  to  10 

Z  lO  1U 

attitudes 

School  Subjects  Attitude  Scales 

students 

5  to  10 

local  surveys 

students 
parents 

administrators 

3  to  10 
2  to  10 

2  to  10 

random  interviews 

students 
parents 
teachers 
support  staff 
administrators 

3  to  10 
3  to  10 
2  to  10 
2  to  10 
2  to  10 
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Statements  to  which  students  responded  included: 

•  I  like  this  process  of  learning. 

♦  This  method  has  helped  me  to  get  higher  math  scores. 

Stakeholders  were  invited  to  provide  additional  comments.  Some  responses  were 
quite  elaborate  and  provided  some  valuable  "thick  description". 

Interviews  -  Random  interviews  were  held  periodically  with  various  stakeholders 
and  followed  a  format  similar  to  the  survey.  Many  of  these  interviews  were  held  in 
the  first  year.  Early  feedback  allowed  us  to  make  appropriate  and  immediate 
changes  in  the  process. 

School  Subjects  Attitude  Scales  -  This  instrument  (Nyberg  &  Clarke,  1983)  provided 
external  quantitative  data  in  the  affective  area.  The  purpose  was  to  assess  students' 
attitudes  toward  mathematics  over  the  three  years.  Students  provided  responses 
related  to  perceived  difficulty,  usefulness,  and  evaluation  of  mathematics.  The  data 
were  collected  annually  from  students  in  grades  5  to  10. 


The  OBE  Sample 

The  sample  included  the  following  stakeholder  groups  in  one  elementary-junior 
high  and  one  high  school: 

Students  The  OBE  pilot  group  included  approximately  600  grade  3  to 

10  students  in  1990-91  and  approximately  700  grade  2  to  10 
students  in  1991-92. 

Teachers  12-15  teachers  implemented  the  pilot  project  in 

approximately  25  classes  per  year,  depending  on  semester 
scheduling  and  teacher  placements. 

Parents  Approximately  360  parents  attended  out  OBE  awareness  and 

input  sessions.  They  provided  survey  and  verbal  responses 
regarding  support  or  improvement  for  the  process. 

All  parents  of  students  in  the  project  were  surveyed  at  the 
end  of  the  1991  school  year,  and  again  at  the  close  of  the 
1992  school  year.  Given  the  number  of  students  in  the 
programs,  approximately  1,300  surveys  were  distributed  in 
the  second  and  third  years  of  the  project. 

Administrators  Two  principals  and  two  vice-principals  were  involved  in  the 

project  over  the  three  years. 


Procedures 

The  three-year  research  project  consisted  of  the  following  actions: 

Year  One  -  team  meetings  to  research  and  implement  the  OBE  process, 
development  of  the  indicators  framework,  selection  of  measures,  and 
development  of  local  measures 
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Year  Two  -  team  meetings  to  discuss  strategies  and  processes, 
implementation  of  the  principles  of  OBE,  field-testing,  analysis  of  the 
data,  and  team  presentation  at  a  national  OBE  conference 

Year  Three  -  team  meetings  to  determine  how  to  refine  and  improve  the  process, 
field-test  review,  analysis  of  the  data,  and  sharing  the  data 


The  OBE  Process 

The  principles  of  Outcome-Based  Education  (Spady,  1988)  guided  our  team 
discussions  and  decisions  throughout  the  three-year  project.  The  principles 
included: 

Designing  Down  -  Our  purpose  was  to  promote  success  for  all  students.  In  order 
to  promote  success,  we  needed  to  clearly  define  the  broad-based,  significant 
outcomes  which  students  would  need  for  their  future.  What  did  future  trends  tell 
us  students  would  need  to  demonstrate  success  in  the  twenty-first  century?  Then 
we  "designed  down"  to  the  program  outcomes.  The  focus  on  responsibility, 
positive  attitudes  and  competent  demonstrations  of  significant  mathematics 
outcomes  provided  a  meaningful  method  of  organizing  for  instruction. 

Maintaining  a  Focus  on  the  Significant  Outcomes  -  We  needed  to  maintain  a  clear 
focus  on  the  outcomes  of  significance.  As  we  became  wrapped  up  in  some  of  the 
details  of  curriculum  we  needed  to  continuously  consider  what  we  were  asking 
students  to  demonstrate.  Our  collaborative  work  kept  us  on  task  and  brought  us 
back  to  the  "drawing  board"  many  times. 

High  Expectations  -  OBE  teachers  believed  that  their  expectations  for  all  students 
to  perform  at  high  levels  had  a  major  impact  on  student  demonstrations  of  the 
outcomes.  Student  work  needed  to  be  done  and  done  well  to  qualify  for  credit 
In  math,  students  were  expected  to  demonstrate  mastery  by  performing  80%  or 
higher  for  each  program  outcome  in  order  to  receive  credit  for  the  skill.  In  some 
cases,  a  skill  was  seen  as  a  "can  do"  skill,  which  did  not  receive  a  mark  at  all. 
Rather,  a  checklist  was  used  and  this  implied,  if  needed,  a  100%  mastery  level. 
Our  reporting  style  changed  in  the  pilot  process  since  averaging  or  summarizing 
a  lot  of  learning  did  not  adequately  describe  what  a  student  could  do.  Reports 
included  a  listing  of  all  the  mathematics  concepts  the  student  had  mastered.  It  also 
included  a  listing  of  the  concepts  which  a  student  had  not  demonstrated  yet,  or  were 
"in  progress". 

Expanded  Opportunities  -  Teachers  were  astutely  aware  that  students  learn  in 
different  ways  and  at  different  rates.  However,  they  were  cognizant  that  the 
opportunities  provided  for  students  to  demonstrate  their  learning  were  frequently 
cut  short  by  time  or  by  the  methods  used  in  the  initial  instruction. 

Teachers  began  to  provide  more  than  one  chance  for  students  to  learn  a  concept 
or  skill.  Ideally,  students  had  the  opportunity  to  participate  in  a  variety  of  active 
learning  experiences  during  the  introduction  of  a  concept.  A  reteaching  process, 
when  needed,  provided  another  opportunity  for  learning.  The  reteaching  might 
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include  such  practices  as  peer  tutoring  or  cooperative  learning,  if  these  were  not 
included  in  the  initial  learning  opportunity.  Additional  opportunities  to  learn 
provided  the  support  many  students  needed  to  demonstrate  success. 

The  issue  of  providing  appropriate  time  for  students  to  demonstrate  their  learning 
was  magnified  by  the  calendar  system.  Some  students  needed  time  beyond  a 
defined  semester  or  school  year  to  demonstrate  the  outcomes.  Similarly,  some 
students  who  mastered  concepts  quickly  needed  the  opportunity  to  continue  their 
progress. 

Dealing  with  the  issues  of  time  and  support  for  learning  required  a  collaborative  and 
creative  vision. 


Findings 

Student  Achievement 

Teacher  Tests  -  Our  findings  indicated  that  many  more  students  were  reaching 
mastery  (80%  or  higher)  levels  in  mathematics  outcomes  than  previously  on 
teacher-developed  assessments.  Figure  1  presents  the  results. 

There  were  substantial  increases  in  achievement  at  the  higher  levels  in  both  Math  10 
and  Math  13  in  1991  and  1992.  Teachers  believed  that  a  clear  focus  on  what  was  to 
be  demonstrated  and  high  expectations  for  all  students  were  making  a  difference  in 
student  performance.  Teachers  however  were  concerned  about  the  students  who 
were  still  reticent  in  accepting  responsibility  for  learning.  Teachers  saw  students 
counting  on  several  chances  to  demonstrate  success  and  procrastinating  in 
demonstrating  the  outcomes.  Time  was  an  obvious  factor  for  teachers  and  students 
in  the  semestered  system.  Reporting  times  were  tedious  for  teachers,  as  students 
attempted  to  "catch  up"  on  demonstration  of  the  outcomes.  In  1992, 19  students 
from  the  Math  10  and  13  classes  withdrew  to  attend  the  school's  personalized 
learning  centre  or  Math  14  classes.  Teachers  felt  these  students  exhibited  behavior 
problems  due  to  their  lack  of  prerequisite  skills  or  understanding  of  the  concepts. 


Percent 


■  Math  10 
□  Math  13 


PreOBE  1990  OBE 1991  OBE 1992 


Figure  1 :  Percentage  of  Mathematics  10  and  13  Students  Achieving 
Mastery  (80%  or  Higher)  on  Teacher  Tests 
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The  failure  rates  for  Math  10  were  26%,  17%  and  31%  respectively  from  1990  to 
1992.  In  Math  13  the  failure  rates  went  from  37%  to  18%  to  26%,  1990  to  1992. 
Teachers  indicated  that  students  did  not  drop  out  of  school  but  rather  made 
decisions  to  get  the  prerequisite  skills  through  another  course  before  registering  for 
Math  10  for  a  second  time.  The  personalized  learning  centre  was  an  attempt  to 
provide  another  learning  opportunity  for  these  students  prior  to  their  enrolling  in 
Math  10.  This  action  corresponded  to  Radwanski's  (1987)  suggestion  that  a 
proactive  process  through  extensive  special  help  would  assist  these  students' 
learning  and  prevent  their  dropping  out  of  school. 

A  review  of  the  first  semester  1992-93  grade  10  math  results  indicated  that  80%  of 
the  total  grade  10  students  who  had  participated  in  two  years  of  OBE  instruction 
(grades  8  and  9)  were  performing  at  mastery  levels  of  80%  or  higher  in  all  math 
courses  in  the  first  term.  Thirty-nine  percent  of  the  OBE  students  who  registered 
for  math  courses  in  the  first  semester  enrolled  in  Math  10.  Of  these,  96%  were 
performing  at  mastery  levels.  Forty-six  percent  of  the  students  took  Math  13;  85% 
of  these  students  were  performing  at  mastery  levels.  Fourteen  percent  of  the  OBE 
students  enrolled  in  nonOBE  math  courses  (Math  14  and  Math  16).  Of  these,  25% 
were  performing  at  the  mastery  levels  in  the  first  term  report. 


Provincial  Achievement  Tests  -  Acceptable  Standard  -  OBE  students  demonstrated 
high  achievement  on  external  measures,  such  as  the  provincial  mathematics 
achievement  tests  for  grades  3, 6  and  9;  85%  of  students  are  expected  to  achieve 
the  acceptable  standard  or  higher  on  the  test  In  all  cases,  OBE  results  showed 
increased  achievement  over  time. 

grade  3 

In  1990, 98%  of  the  OBE  grade  3  students  achieved  the  acceptable  standard  on  the 
provincial  achievement  test.  This  performance  was  repeated  in  1991.  In  1992  all 
grade  3  students  were  performing  at  the  acceptable  standard.  Figure  2  displays  the 
results.  It  may  be  argued  that  the  teachers  were  teaching  to  the  test.  Teachers 
valued  the  conditions  of  the  provincial  testing  process  and  the  school  and  district 
attempted  to  maintain  these  conditions  at  all  times.  Teachers  were  dedicated  to 
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Figure  2 :  Percentage  of  Grade  3  Students  Achieving  the  Acceptable 
Standard  on  the  Total  Provincial  Math  and  Science  Tests 
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teaching  the  curriculum  outcomes.  They  believed  it  was  the  dedication  to  the 
teaching  of  the  outcomes,  and  the  provision  for  time  and  opportunity  to  learn,  that 
caused  these  results  to  occur.  In  fact,  in  1991  grade  3  students  wrote  the  provincial 
science  test  and  teachers  indicated  that  they  spent  time  preparing  for  that  test.  They 
were  keen  to  see  the  results  because  they  felt  there  was  a  great  deal  of  overlap 
between  science  and  mathematics  in  1991,  for  example,  graphing  and  practical 
applications.  They  believed  strongly  that  OBE  transfers  into  other  subject  areas. 

The  grade  3  teachers  shared  the  following  comments  when  asked  for  their 
perceptions  regarding  the  results: 

"We  were  directed  to  think  critically  about  what  the  children  needed  to  do 
easily  and  what  we  could  do  without.  We  took  things  that  would  be 
beneficial  for  student  learning  and  did  that  EARLY  so  the  students  would 
have  the  skills.  Problem  solving  was  taught  early  because  it  was  a  skill  they 
had  to  have.  We  made  them  discuss  and  verbalize  what  should  be  done,  e.g., 
process.  They  [students]  made  up  their  own  problems  for  other  kids  to  do! 
Subsequent  new  skills  in  regrouping  were  taught  in  the  problem  solving 
format.  We  had  students  drawing,  using  manipulatives,  talking  and  sharing 
ideas  with  a  partner.  Students  worked  in  groups  of  4.  We  tried  to  see  who 
could  come  up  with  a  way  to  solve  the  problem.  We  stressed  the 
PROCESS,  not  the  answer,  thereby  reinforcing  the  students'  thought 
process." 

The  grade  3  teachers  expressed  surprise  at  the  survey  results  they  had  collected 
in  their  own  classrooms.  Students  positively  ranked  mathematics  with  physical 
education  and  art.  They  also  expressed  surprise  at  the  students'  reaction  to  the 
writing  of  the  achievement  exam.  "They  love  the  achievement  test!" 

The  1992  results  were  similar.  All  students  performed  at  or  above  the  acceptable 
standard. 


Grades  6  and  9 

In  1991,  the  grade  6  students  performed  slighdy  above  their  local  peers  on  the 
mathematics  achievement  test.  There  was  a  significant  variation  in  collaborative 
efforts  and  implementation  between  classes  at  this  level  compared  to  the  grade  3 
level.  Basically,  collegial  planning  did  not  occur  other  than  involvement  in  the 
district  team  planning  sessions.  The  results  for  the  two  classes  are  combined. 
The  1992  grade  6  OBE  results  showed  higher  performances  than  the  local  and 
provincial  results  of  1991.  Figure  3  presents  the  results. 

Teacher  descriptions  of  the  OBE  process  at  the  grade  6  level  included: 

"My  implementation  of  OBE  changed  the  achievement  results  and  attitudes, 
e.g.,  self-confidence,  of  these  students!" 

"The  principles  transfer  into  all  the  different  subjects." 
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Figure  3:  Percentage  of  Grade  6  and  9  Students  Achieving  the  Acceptable 
Standard  on  the  Total  Provincial  Mathematics  Tests 


Some  of  the  changes  in  practice  included  the  following: 

•  In  1989-90  students  received  few  expanded  opportunities.  In  OBE 
practices,  students  received  more  opportunities. 

•  Previously,  everything  counted,  for  example,  "stuff  students  didn't  know  was 
averaged  in  with  the  "stuff'  they  knew  and  was  held  against  them  at  every 
reporting  period.  In  OBE  practices,  record  keeping  became  very  focused  on 
outcomes.  Students  got  credit  when  they  mastered  an  outcome.  One  teacher 
stated,  "I  didn't  grade  as  much  this  year  as  last  -  only  once  the  students  were 
successful."  Previously,  some  random  record  keeping  occurred.  "The  next 
space  in  the  book  was  the  next  activity  we  did!"  The  record  keeping  system 
became  a  diagnostic  tool  and  identified  student  needs  and  successes. 

In  1991, 60  grade  9  OBE  students  wrote  the  1988  Alberta  Education  Pilot  Math 
Test  Only  28%  of  the  students  performed  at  the  acceptable  provincial  standard. 
Preceding  and  following  the  test,  the  students  indicated  their  perceptions  that  this 
test  was  not  "worth"  any  credit,  so  they  did  not  attempt  to  achieve  their  best.  In 
1992,  during  a  discussion  following  the  writing  of  the  provincial  test,  students 
indicated  that  they  had  tried  their  best.  Some  students  felt  the  questions  had  been 
"tricky".  Their  1992  performance  was  lower  than  the  provincial  average  but 
slightly  higher  than  the  local  results.  The  1992  grade  9  OBE  performance 
showed  a  dramatic  increase  over  the  1991  results. 

Teacher  practices  included  a  great  deal  of  assistance  to  students  after  class  hours 
and  the  choice  for  students  to  move  ahead  beyond  the  regularly  defined  time 
constraints.  In  the  1991-92  school  year,  four  students  completed  the  course 
outcomes  prior  to  the  end  of  the  year  and  moved  to  the  next  grade  level  outcomes 
midyear.  Other  variations  in  inputs  included  additional  preparation  times  for  OBE 
planning  in  mathematics.  A  teacher  aide  was  available  for  math  classes.  Parent 
meetings  provided  awareness  for  parents  and  support  for  the  process. 
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Provincial  Achievement  Tests  -  Standard  of  Excellence  -  The  percentage  of 
students  achieving  the  standard  of  excellence  is  displayed  in  the  following  figures; 
15%  of  students  are  expected  to  achieve  the  standard  of  excellence  on  the  test. 
In  all  cases  where  repeated  tests  were  administered,  results  over  time  showed 
improvement  gains  in  the  OBE  classes,  as  compared  to  earlier  results. 

In  1992,  a  greater  percentage  of  the  grade  3  OBE  students  reached  the  standard  of 
excellence  in  mathematics  than  in  1990.  This  was  lower  than  the  1991  results  but  a 
significant  increase  from  1990.  The  OBE  results  in  science  were  consistent  with 
the  mathematics  results,  although  the  percentage  of  students  performing  at  the 
standard  of  excellence  was  lower.  OBE  students  outperformed  their  local  and 
provincial  peers  in  science.  The  differences  between  grade  3  OBE  and  provincial 
results  were  significant  Figure  4  presents  the  results. 
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Figure  4:  Percentage  of  Grade  3  Students  Achieving  the  Standard  of 
Excellence  on  Total  Provincial  Mathematics  and  Science 
Achievement  Tests 
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Figure  5:  Percentage  of  Grade  6  and  9  Students  Achieving  the  Standard  of 
Excellence  on  the  Total  Provincial  Mathematics  Achievement  Tests 
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The  1992  grade  6  results  showed  a  dramatic  increase  in  mathematics  achievement 
over  the  results  of  1991.  The  1992  grade  9  OBE  students  did  not  match  the 
provincial  or  local  performances  in  the  standard  of  excellence  but  they  increased 
their  own  school  performance  slightly  over  the  1991  results.  Figure  5  presents 
the  results  for  grade  6  and  9  students. 


Stakeholder  Perceptions  Regarding  Outcomes 

Parent  and  student  surveys  showed  perceived  gains  in  student  achievement, 
responsibility  and  attitudes  as  a  result  of  the  OBE  process.  Although  the  results 
were  positive,  the  number  of  parents  responding  was  low  (N=98, 15%  response 
rate  in  1991;  N=150, 19%  response  rate  in  1992,).  Teacher  responses  indicated  a 
perceived  decrease  in  positive  changes  although  interviews  with  teachers  indicated 
that  expectations  for  the  demonstrations  of  the  outcomes  were  higher  in  1992. 
Figures  6  through  8  present  the  results. 

Teacher  perceptions  suggest  that  we  may  have  experienced  the  Hawthorne  effect. 
In  1991,  all  teachers  indicated  a  substantial  change  in  student  achievement, 
responsibility  and  attitudes.  Interviews  with  teachers  in  1992  indicated  their 
continued  high  expectations  for  student  achievement.  However,  teachers  believed 
achievement  could  be  higher,  students  attitudes  could  be  more  positive  and 
students  could  show  greater  responsibility  for  their  own  learning.  In  each  area, 
achievement,  responsibility  and  attitudes,  teacher  perceptions  moved  from  100% 
in  1991  to  80%  in  1992. 

The  most  significant  increase  in  stakeholder  perceptions  appeared  in  the  area  of 
student  responsibility  as  perceived  by  parents.  One  parent  indicated  in  1991  that 
her  child  was  even  choosing  to  clean  up  her  room  now.  The  mother  attributed  this 
phenomenon  to  OBE,  believing  it  had  made  her  child  feel  better  about  herself, 
thereby  increasing  her  desire  to  demonstrate  responsibility  at  home.  Student 
responses  related  to  perceived  increases  in  responsibility  were  also  higher  in  1991. 
Teachers  felt  students  could  demonstrate  greater  responsibility  for  learning. 

Parent  responses  regarding  perceived  increases  in  positive  student  attitudes  were 
higher  in  1992  than  in  1991.  A  slight  increase  in  student  perceptions  was  noted. 
Teachers  felt  students  could  be  more  positive. 

Whether  students  feel  better  about  themselves  as  a  result  of  achievement  or  whether 
they  need  to  feel  good  about  themselves  in  order  to  achieve  well  has  long  been 
debated.  The  results  appear  to  suggest  a  relationship  between  achievement  and 
gains  in  the  affective  and  behavioral  domains. 
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Figure  6:  Percentage  of  Stakeholders  Indicating  Positive  Changes 
in  Student  Achievement  in  Mathematics 
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Figure  7:  Percentage  of  Stakeholders  Indicating  Positive  Changes 
in  Student  Responsibility 
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Figure  8:  Percentage  of  Stakeholders  Indicating  Positive  Changes 
in  Student  Attitudes 
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The  School  Subjects  Attitude  Scales  (Nyberg  &  Clarke,  1983)  were  administered 
annually  over  the  three  years.  Table  2  presents  the  results.  A  review  of  the  results 
led  to  a  suggestion  that  since  the  means  did  not  vary  significantly,  only  minimally  if 
at  all,  interpretations  should  be  made  with  caution.  Given  these  cautions,  we  make 
the  following  observations.  There  appeared  to  be  some  positive  gains  in  the  grades 
5  and  6  OBE  classes  while  district  results  have  stayed  virtually  the  same  over  the 
three  years.  Grade  7  to  9  OBE  students  appeared  to  think  math  is  less  useful  than 
previously.  OBE  Math  10  results  suggest  that  students  think  math  is  less  useful 
than  previously.  OBE  Math  13  results  suggest  that  some  students  believe  math  is 
more  useful  and  they  appear  to  like  it  slightly  more  than  previously.  Grade  7  to  10 
students  appear  to  think  math  is  not  an  easy  subject. 


Table  2 


Mean  Scores  for  Attitudes  Toward  Mathematics  on 
the  School  Subjects  Attitude  Scales 


Mathematics      School  District 

PreOBE 

OBE 

OBE 

Attitudes 

1990 

1991 

1992 

1990 

1991 

1992 

urducs  jmo 

\1> — D\rJJ 

— DODJ 

(N=84) 

(N=97) 

(N=102) 

Evaluation 

27.5 

26.6 

27.6 

21.3 

27.6 

28.5 

Usefulness 

33.9 

33.8 

33.8 

H  6 

35  0 

JJ<U 

Difficulty 

26.3 

25.3 

26.3 

24.6 

26.8 

27.1 

Grades  7-9 

(N=434) 

(N=522) 

(N=461) 

(N=183) 

(N=243) 

(N=229) 

Evaluation 

23.9 

22.7 

29.5 

24.4 

24.1 

24.6 

Usefulness 

32.9 

32.3 

34.1 

34.4 

33.1 

31.8 

Difficulty 

22.0 

23.1 

22.9 

21.4 

21.4 

21.9 

Math  10 

(N=102) 

(N=65) 

(N=80) 

Evaluation 

23.9 

20.2 

21.4 

Usefulness 

34.5 

31.5 

26.9 

Difficulty 

19.7 

20.5 

18.1 

Math  13 

(N=37) 

(N=44) 

(N=27) 

Evaluation 

21.7 

25.0 

23.1 

Usefulness 

28.6 

33.3 

28.9 

Difficulty 

23.6 

23.0 

21.8 

Note.  A  mean  score  of  24  represents  a  neutral  score  whereas  a  mean  above  24  represents  a  positive 
feeling  toward  a  subject  and  one  below  24  a  negative  attitude. 
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Parental  Support  for  the  OBE  Process 

Parental  support  for  the  Outcome-Based  Education  process  increased  at  the  grade  2 
to  9  levels  in  the  second  year  of  implementation.  There  was  a  decrease  of  support 
at  the  grade  10  level.  Figure  9  presents  the  results.  Generally,  parental  support  for 
the  OBE  process  was  very  positive.  A  significant  increase  in  support  occurred  at 
the  junior  high  levels.  A  decrease  was  evident  at  the  grade  10  level.  At  a  meeting 
with  parents  of  grade  10  students,  parents  indicated  they  were  supportive  of  the 
philosophy  and  principles  of  OBE  but  they  believed  that  our  "time  to  learn" 
principle  was  not  appropriately  adhered  to  through  the  semester  system.  This 
constraint  had  also  been  noted  by  teachers.  It  was  hoped  that  technological  systems 
would  support  and  improve  time  to  learn  practices.  An  integrated  learning  system 
is  currently  under  district  review. 

Table  3  presents  stakeholders'  perceptions  about  the  strengths  and  weaknesses  of 
the  OBE  approach. 


■  Very  Positive 
□  Moderate 

H  Not  Supportive 

■  No  Response 


Grades  2  to  6       Grades  7  to  9  Grade  10 


Figure  9:  Percentage  of  Parents  Indicating  Support  for  the  OBE  Process 


Conclusions 

There  is  strong  quantitative  and  qualitative  evidence  that  the  OBE  process  produces 
positive  results  in  student  achievement,  responsibility  and  attitudes  in  mathematics. 

The  improved  outcomes  appear  to  have  been  brought  about  by  teachers  focusing  on 
the  principles  of  OBE.  The  principles  include  designing  down  from  the  significant 
outcomes,  maintaining  a  clear  focus  on  the  outcomes,  expanded  time  and 
opportunity  for  learning,  and  high  expectations  for  the  success  of  all  students. 

Our  results  indicate  that: 

•  More  students  are  achieving  at  higher  levels  of  performance  on  teacher 
developed  and  external  measures. 

•  There  was  an  increase  in  the  percentage  of  parents  showing  support 
for  the  process  at  elementary  and  junior  high  levels. 
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Table  3 


Stakeholder  Perceptions  Regarding  Outcome-Based  Education 


Stakeholder 

Strengths 

Weaknesses 

Parents 

•  high  standards  for  achievement 

•  high  expectations  for  students  to  work 
at  it  and  perform  well 

•  process  builds  self-esteem  and  confidence 

•  second  chances  are  realistic 

•  promotes  a  solid  understanding  for  math 

•  the  clear  reporting  system 

•  teacher  support  for  student  success 

•  students  who  move  to  nonOBE  classes 
do  not  necessarily  perform  well  in  those 
classes  even  with  having  had  time  in  the 
OBE  process 

•  second  chances  are  not  realistic 

•  opportunity  to  "goof  off  and  not  try 
hard  the  first  time 

•  semester  system  goes  too  fast 

Students 

Grades 
3  to  6 

•  OBE  is  less  confusing  and  easier 

•  second  chances  help  to  improve 
math  skills 

•  useful  to  get  a  job 

•  it  provides  a  challenge 

•  it's  fun 

•  it's  a  bother,  sports  activities  may  be 
missed 

•  teachers  expect  too  much 

•  the  process  continues  even  when 
students  are  not  finished 

Grades 
7  to  9 

•  more  is  learned  and  remembered 

•  allows  students  to  learn  from  mistakes 

•  students  feel  they  actually  learn  the 
concepts 

•  helps  students  become  more  responsible 

•  will  provide  opportunity  to  get  a  good 
job  and  be  proud  of  self 

•  students  can  get  behind 

•  second  chances  are  not  realistic 

•  too  many  outcomes  in  a  year 

Grade  10 

•  the  process  is  easier  and  students  are 
confident  about  passing 

•  better  learning  occurs 

•  increases  understanding  in  math  even 
after  tests  have  been  written 

•  less  tension  when  you  write  a  test 

•  OBE  Math  is  no  longer  a  difficult 
subject  to  grasp 

•  students  feel  rushed  but  like  the 
second  chances 

•  it's  demanding 

•  80%  expectation  is  too  high 

Teachers        •  instructional  clarity  •  how  to  improve  student  responsibility 

•  collaboration  and  higher  level  thinking  skills 

•  focus  on  quality  •  the  need  to  show  students  how  math  is 

•  diagnostic  emphasis  used  in  the  day-to-day  world 

•  student  confidence  •  how  to  deal  with  the  OBE  principles 

•  love  of  learning  within  a  semestered  system 


Source:  Stakeholder  Surveys,  Fort  McMurray  Catholic  Schools 
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♦    There  were  overall  gains  in  parent  and  student  perceptions  regarding 
positive  changes  in  student  achievement,  responsibility  and  attitudes. 

The  OBE  improvement  process  evolved  over  time  through  collaborative  visioning 
and  efforts.  The  team  heeded  the  caution  that  change  occurs  over  time  and  that 
sufficient  time  must  be  allocated  to  encourage  long-term  gains  in  ownership  and 
results. 


Discussion 

The  findings  imply  that  the  OBE  process  provides  a  successful  framework  for  the 
delivery  of  a  variety  of  effective  instructional  practices.  The  results  of  the  OBE 
practices  show  improved  student  achievement,  responsibility  and  attitude  outcomes. 

Districts  are  invited  to  review  the  principles  of  OBE  and  determine  what  might  be 
useful  in  their  journey  toward  improved  student  outcomes.  We  hope  that 
jurisdictions  will  take  the  best  from  the  lessons  we  have  learned  and  make 
adaptations  appropriate  to  their  local  needs. 

Stakeholders  might  ask,  "If  OBE  promotes  student  success,  why  aren't  more 
schools  using  it?"  As  a  result  of  the  requests  for  information  we  have  received,  it  is 
clear  that  stakeholders  all  across  Canada  are  keenly  interested  in  the  OBE  process. 
There  is  a  trend  toward  defining  what  it  is  we  want  students  to  demonstrate  in  the 
future  and  what  we  need  to  do  now  to  promote  student  success. 

Our  experiences  with  OBE  have  created  local  ownership  of  Vision  for  the  Ninettes 
(Alberta  Education,  1991).  We  encourage  Alberta  Education  to  continue  the 
emphasis  on  educational  improvement  by  focusing  on  outcomes. 

Alberta  Education  has  modeled  the  collaborative  improvement  process  through  the 
Educational  Quality  Indicators  initiative.  We  believe  the  province's  collaborative 
focus  on  outcomes  is  a  positive  one.  Local  indicators  provide  a  picture  of  the 
general  well-being  of  education  in  classrooms  and  schools.  The  use  of  a  variety  of 
indicators  can  provide  a  significant  and  varied  source  of  information  from  which 
teachers  and  administrators  can  make  informed  decisions  regarding  educational 
improvement. 

A  limitation  was  the  absence  of  reliable  and  valid  mathematics  achievement  data 
which  would  have  allowed  us  to  make  annual  comparisons  with  students  across 
Alberta.  Our  project  would  have  benefited  from  such  a  provincial  process.  In 
particular,  annual  performance  measures  to  assess  higher  level  thinking  skills 
would  have  been  most  helpful. 


Follow-up 

The  EQI  project  has  officially  concluded  but  the  teachers  involved  in  the  project 
see  the  need  for  a  long-term  improvement  vision.  They  are  committed  to  the 
continuation  and  improvement  of  the  OBE  process. 
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Suggestions  for  Improvement 

Teachers  and  parents  have  made  the  following  suggestions  for  improving  the  OBE 
approach. 

Teachers: 

•  greater  encouragement  for  students  to  move  beyond  regular  calendar-defined 
"grade"  levels 

•  continuation  of  the  design  down  work 

•  resource  availability  in  each  classroom  to  promote  learning  beyond  designated 
grade  levels 

•  greater  integration  of  higher  level  thinking  opportunities  through  instruction 
and  assessments 

•  technological  support  for  continuous  reporting 

•  continued  collaborative  efforts  in  planning 

Parents: 

•  teacher  training  emphasis  in  motivation  techniques  for  today's  generation 

•  expand  the  program  to  grades  1 1  and  12,  to  other  subjects  and  schools 

•  track  the  number  of  times  students  take  to  complete  an  outcome 

•  choices  regarding  participation  in  the  process 

•  fun  math  and  role  playing,  e.g.,  banker,  farmer,  stock  market,  etc. 

•  more  time  in  class  to  complete  outcomes 

•  change  the  standard  to  90%;  70% 

•  elirninate  deadlines  for  demonstrations  of  outcomes 

•  have  fewer  outcomes 


Other  OBE  Support  Strategies 

In  addition  to  a  board  resolution  to  continue  the  expansion  of  the  OBE  principles  in 
the  district,  the  focus  on  student  success  is  a  major  component  of  the  district's  new 
Strategic  Plan.  This  plan,  which  has  had  wide  representation  from  key  stakeholder 
groups,  is  expected  to  significantly  impact  education  in  the  Fort  McMurray  Catholic 
Schools.  The  following  outcomes  were  identified  as  significant  demonstrations  of 
success  required  by  graduates  of  our  district: 

•  mastery  of  Program  of  Studies  outcomes 

•  Catholic  Christian  service  and  witness 

•  problem  solving  and  decision  making 

•  critical  and  creative  thinking 

•  learning  to  learn 

•  positive  attitudes/behaviors,  responsibility,  adaptability 

•  effective  communication 

•  cooperation 

•  continuing  to  learn 

•  self-esteem 

Emphasis  will  be  placed  on  defining  these  outcomes  of  significance  and  developing 
instructional  strategies  and  data  collection  procedures  for  each  of  the  outcomes. 

A  collaborative  review  of  current  indicators  and  needs  related  to  the  outcomes  is 
occurring  through  School  Improvement  Teams.  These  teams  are  comprised  of 
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parents,  administrators,  teachers  and  students.  Based  on  their  assessment  of 
current  results  and  needs,  the  teams  will  collaboratively  plan  for  improvement 
to  enhance  student  success. 

School  Management  Plans  will  be  focused  on  the  improvement  of  delivery  and 
results  in  the  district's  outcomes  of  significance.  The  management  plans  are 
developed  annually  by  administrators  and  teachers  at  each  school. 


Concluding  Statement 

Outcome-based  education  is  a  school  improvement  and  restructuring  process;  it  is 
not  a  program.  Collaborative  teacher  efforts  are  critical  to  the  effectiveness  of  our 
improvement  process.  Teaching  teams  may  implement  the  process  in  slightly 
different  ways  but  the  questions  all  teams  should  ask  are: 

1 .  Are  students  learning  what  we  believe  is  important  to  learn? 

2.  What  evidence  do  we  accept  that  students  are  demonstrating  success  in  the 
outcomes  of  significance? 

3 .  Based  on  our  understanding  of  results,  what  can  we  do  to  improve  the 
instructional  conditions  so  students  learn  and  progress  continuously? 

The  premises  and  principles  of  OBE  have  provided  the  foundation  for  our 
collaborative  improvement  process  over  the  last  three  years.  The  indicators 
strongly  imply  that  the  outcome-based  process  implemented  within  a  collaborative 
environment  has  produced  positive  changes  in  student  achievement,  responsibility 
and  attitudes. 

The  OBE  process  is  a  positive  way  to  promote  student  success.  We  believe  that 
many  of  our  students  will  be  better  able  to  cope  academically  in  high  school  because 
they  have  arrived  from  elementary  school  better  prepared.  We  also  believe  the  OBE 
process  will  better  prepare  our  students  for  success  after  they  leave  our  schools 
because  they  have  demonstrated  greater  success  in  the  outcomes  of  significance. 


References 

Alberta  Education.  (1991).  Vision  for  the  nineties  ...  a  plan  of  action.  Edmonton: 
Author. 

Nyberg,  V.R.,  &  Clarke,  S.C.T.  (1983).  School  subjects  attitude  scales. 
Edmonton:  Planning  Services  Branch. 

Radwanski,  G.  (1987).  Ontario  study  of  the  relevance  of  education,  and  the 
issue  of  dropouts.  Toronto:  Ontario  Ministry  of  Education. 

Spady,  W.  G.  (1988).  Organizing  for  results:  The  basis  of  authentic 
restructuring  and  reform.  Educational  Leadership,  46(2),  4-8. 


OBE  Project 


79 


Fort  McMurray  Catholic 


Student  Growth:  The  Development  of 
Enhanced  Practices  for  Assessment, 
Evaluation,  and  Communication 


Edmonton  School  District  No.  7 


This  report  of  work  in  progress  provides  details  of  Edmonton  Public  Schools' 
activities  and  findings  in  the  developmental  work  carried  out  from  1989  to  1992 
related  to  the  Alberta  Education  "Educational  Quality  Indicators"  initiative.  It 
traces  the  dynamic  shifts  in  developmental  focus  from  simple  identification  of 
generic  indicators  and  measures  of  student  growth  to  coordinated  development 
of  mechanisms  such  as  "portfolio  assessment",  the  use  of  computer  technology 
for  communication  and  storage  of  information  about  student  growth,  and  the 
establishment  and  validation  of  curriculum-based  district  level  assessment  strategies 
from  grades  1  to  12.  The  report  presents  the  assumptions  and  the  developmental 
rationale  on  which  the  interpretation  of  the  results  is  based.  The  district  is 
continuing  its  initiatives  in  this  domain.  The  planned  direction  of  future 
developmental  work  is  outlined. 


Introduction 

Edmonton  Public  Schools  identified  as  a  first  requirement  that  a  review  be  carried 
out  of  existing  indicators  and  measures  of  district  performance  used  in  the  ongoing 
performance  monitoring  processes.  At  that  time,  these  included  the  results  of: 

•  annual  staff  and  student  district  attitude  surveys 

•  biennial  parent  and  general  community  attitude  surveys 

•  district  grade  3, 6  and  9  achievement  tests  in  mathematics, 
language  arts,  science  and  social  studies 

•  Alberta  Education  grade  3,  6,  and  9  achievement  tests 

•  Alberta  Education  grade  12  diploma  examinations 

•  International  Baccalaureate  examinations 

Although  these  indicators  and  measures  provided  useful  information  about  the 
district's  overall  performance,  there  were  obvious  deficiencies  in  the  range  and 
nature  of  student  achievement  strategies  and  measures.  The  curriculum-based 
tests  were  unable  to  differentiate  among  grade  levels  of  achievement  except  at  the 
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3,  6, 9,  and  12  levels.  The  usefulness  of  this  type  of  externally  evaluated  "paper 
and  pencil"  testing  at  widely  dispersed  grade  levels  was  considered  inadequate. 
The  tests  were  also  unable  to  extract  from  these  results  any  measures  of  student 
performance  related  to  six  essential  learning  outcomes  established  as 
communication,  responsible  citizenship,  well-being,  knowledge,  inquiry,  and 
aesthetic  appreciation. 

To  achieve  the  result  of  the  project  that  students,  parents,  staff,  the  board,  and  the 
general  community  have  valid,  accurate,  and  useful  information  about  student 
achievement  and  growth,  the  following  project  requirements  were  established: 

•  instruments  and  strategies  that  will  facilitate  assessment  and  annual 
communication  of  student  achievement  for  individual  student,  school  and 
district  reporting  requirements  are  developed 

•  standards  against  which  student  achievement  is  evaluated  relative 
to  the  graded  curriculum  are  established 

•  assessment,  evaluation,  and  communication  of  continuous 
development  of  skills,  attitudes,  and  knowledge  are  emphasized 

•  strategies  are  designed  to  gather  teacher  judgments  and  evidence 
of  student  demonstration  of  the  curriculum  relative  to  the 
essential  learning  outcomes 

•  opportunity  is  provided  for  students  to  demonstrate  their  highest 
level  of  achievement 


Rationale 

The  assessments  that  count  -  the  assessments  that  most  strongly 
influence  student  learning  and  academic  self-concept  -  are  those 
developed  and  used  by  teachers  in  the  classroom  (Stiggins,  1988). 

1 .  Legal  requirements  established  in  Section  13  of  the  School  Act: 

♦  The  student's  teacher  is  responsible  for  assessing  and  periodically 
reporting  student  achievement. 

2.  Assumptions: 

♦  The  achievement  and  performance  of  each  student  is  measurable. 

♦  Student  growth  in  achievement  is  one  of  the  indicators  for  measuring 
student,  school  and  district  performance. 

♦  Teacher  judgment  about  student  achievement  and  performance  is  valid 
and  reliable. 
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3.  Principles: 

•  Assessment,  evaluation,  and  communication  of  student  growth  are  based 
on  the  curriculum  and  are  aligned  with  the  school's  philosophy  and 
programming  principles. 

•  Information  about  methods  of  assessment  and  results  of  evaluation  is 
available  to  students,  parents  and  the  community. 

•  Student  growth  is  assessed,  evaluated,  and  communicated  for  all  learning 
outcomes. 

•  Evaluation  and  communication  of  student  growth  is  ongoing  and  used  to 
plan  effective  programming. 

•  Student  growth  is  demonstrated  through  a  variety  of  performances 
evaluated  by  the  teacher. 

•  Student  growth  is  enhanced  when  students  participate  in  the  assessment, 
evaluation,  and  communication  processes. 

•  Student  growth  is  enhanced  when  assessment,  evaluation,  and 
communication  are  viewed  positively  by  the  student 

•  Methods  of  communicating  student  growth  vary  depending  on  audience 
and  purpose. 

•  Methods  of  assessment  and  evaluation  of  student  growth  vary  depending 
on  student  learning  patterns  are  developmentally  appropriate. 


Design 

To  achieve  the  planned  results  of  the  developmental  project,  the  framework 
assumed  the  following  major  components: 

1 .  design  and  validation  of  instruments  and  strategies  to  allow  for  student 
demonstration  of  achievement; 

2 .  design  and  validation  of  strategies  and  criteria  to  allow  for  teacher  judgment, 
data  collection,  and  confirmation; 

3.  investigation  of  workable  applications  of  the  "portfolio  assessment"  concept; 

4 .  development  of  prototype  technology  that  would  accommodate  school  and 
district  needs  for  computer  display,  communication,  and  storage  of  information 
about  student  achievement  and  growth;  and 

5 .  overall  management  of  the  project  to  ensure  coordination  of  the  four 
developmental  areas. 
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Measuring  Growth 

To  enhance  current  district  and  provincial  assessment  practices  and  to  overcome 
the  deficiencies  of  the  conventional  paper  and  pencil  "fill  in  the  blank  or  circle  the 
correct  answer"  type  of  assessment  instrument,  emphasis  was  placed  on  the  concept 
of  multiple  activity  tasks  linked  to  the  essential  learning  outcomes. 

Requirements  for  the  design  of  an  initial  task  for  field-testing  were  established: 

•  Tasks  are  seen  to  be  valid  and  reliable  by  students,  staff,  parents,  and  the 
community  and  these  publics  are  satisfied  with  the  information  they  are  supplied 
about  student  growth. 

•  The  tasks  are  based  on  the  essential  learning  outcomes. 

•  Teachers  ate  involved  in  the  development  of  criteria  and  the  growth  measure. 

•  Criteria  are  developed  to  evaluate  the  grade  level  of  achievement. 

•  The  tasks  are  undertaken  by  students  currently  demonstrating  a  wide  range  of 
performance  and  achievement. 

•  The  tasks  are  as  typical  as  possible  of  normal  classroom  and  real-life 
experiences  and  involve  the  students  in  multiple  activities. 

•  There  is  opportunity  for  student-to-student  and  teacher-to-student  interactions. 

•  Classroom  teachers  are  trained  to  use  student  products  in  making  judgments 
about  student  achievement. 


Portfolio  Assessment 

One  of  the  ways  of  enhancing  current  practices  for  assessing,  evaluating,  and 
communicating  student  achievement  and  growth  is  through  the  use  of  portfolios. 
To  guide  the  development  work  on  this  topic,  Edmonton  Public  Schools  has 
defined  a  "student  portfolio"  as  a  purposeful  collection  of  student  products  that 
exhibits  to  the  student  and  others  the  student's  demonstration  of  achievement, 
growth  and  performance  in  essential  learning  outcomes.  This  collection  includes 
criteria  for  selection,  achievement  criteria,  student  self-reflections,  and  teacher  notes 
describing  the  context  within  which  the  products  were  obtained.  The  student  is 
involved  in  the  selection  of  the  contents. 

The  relationship  of  the  use  of  student  portfolios  to  a  generally  enhanced  school  and 
district  assessment  system  can  be  illustrated  by  the  conceptual  schema  shown  in 
Figure  1. 
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PORTFOLIO  ASSESSMENT  SYSTEM 
A  CONCEPTUAL  SCHEMA 
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Figure  1 :  Portfolio  Assessment  System 
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The  relationship  of  the  portfolio  technique  to  developmental  assessment  is  aptly 
described  by  David  Elkind: 

Developmental  assessment  involves  documenting  the  work  that  a  child  has  done 
over  a  period  of  time.  Usually  this  is  done  by  having  a  child  keep  a  portfolio  that 
includes  all  of  his  or  her  writing,  drawing,  math,  explorations,  and  so  on.  In  looking 
through  such  a  portfolio,  we  can  get  a  good  idea  of  the  quality  of  work  that  the  child  is 
capable  of  doing  and  of  his  or  her  progress  over  a  given  period  (Elkind,  1989). 

The  initial  developmental  work  on  the  student  portfolio  focused  on  the  following 
issues: 

•  Application  -  the  usefulness  of  the  information 

•  Responsibility  -  the  role  of  students,  teachers,  parents,  community, 
administrators,  and  trustees 

•  Content  -  what  is  developed  and  how 

•  Design  -  task  development,  validation,  and  administration 

•  Evaluation  -  the  criteria  for  making  judgments 

•  Management  and  Logistics  -  storage,  movement,  access,  School  Act 
requirements  for  student  records 

•  Implementation  -  staff  development  and  training 

•  Reporting  -  communication  of  student  growth  information  to  the  many  publics 

The  development  of  workable  strategies  for  teacher  use  of  portfolio  assessment  in 
the  day-to-day  realities  of  the  classroom  environment  was  seen  to  take  priority  over 
any  concerns  about  the  physical  format  of  a  portfolio.  Differentiation  had  to  be 
made  between  the  use  of  portfolio  contents  for  district  accountability  and  for 
individual  student  assessment. 

There  has  been  extensive  teacher  involvement  in  the  ongoing  development  of 
appropriate  teacher  strategies  and  in  the  identification  of  a  realistic  set  of  student 
products  that  could  be  incorporated  within  the  portfolio.  The  following  have  been 
identified  for  further  investigation: 

•  narrative,  personal,  and  informational  writing 

•  student  self-reflection  analyses 

•  photographs,  videocassettes,  and  audiotapes 

•  letters  of  reference 

•  tables  and  reports  of  investigative  student  work 

•  models 

•  artwork  of  all  kinds 

•  student  generated  computer  disks 

•  teacher  judgments,  assessment  criteria,  task  descriptions,  exemplars,  and 
context  notes 
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Prototype  Computer  Technology 


In  the  initial  project  design,  it  was  clear  that  there  would  be  two  major  logistical 
restraints  to  obtaining  maximum  benefit  from  the  proposed  enhanced  student 
assessment  process: 

•  human  limitations  on  the  part  of  the  teacher  in  coping  with  a  wide  array  of 
media  and  achievement  criteria;  and 

•  communicating  student  growth  information  to  parents  and  aggregated  school 
and  student  information  to  the  community. 

The  design,  therefore,  assumed  the  following  objectives: 

•  to  investigate  the  potential  of  current  "pen-based"  computer  technology 

•  to  develop  prototype  user  screens  for  entry  and  presentation  of  student  growth 
information 

•  to  identify  requirements  for  upgrading  the  wide  area  district  network  to 
accommodate  the  bandwidth  required  for  central  collection  of  student 
achievement  data,  including  compressed  image  data  transfer 

•  to  develop  teacher  usable  applications  for  local  computer  preparation  of 
meaningful  "progress  reporting"  to  parents 


Findings 

To  solely  use  a  standardized  achievement  test  is  like  casting  a  net  into  the  sea 
-  a  net  that  is  intentionally  designed  to  let  the  most  interesting  fish  get  away. 
Then,  to  describe  the  ones  that  are  caught  strictly  in  terms  of  their  weight  is  to 
radically  reduce  what  we  know  about  them.  To  further  conclude  that  all  the 
contents  of  the  sea  consist  of  fish  like  those  in  the  net  compounds  the  error 
further.  We  need  to  know  more  about  those  we  catch.  We  need  new  nets. 

(Randall,  1992) 

It  is  important  to  emphasize  that  the  new  strategies  designed,  field-tested,  and 
piloted  in  this  developmental  project  are  supplements  to  rather  than  replacements 
for,  the  traditional  and  current  assessment  practice.  The  results  demonstrate  the 
potential  for  enhancement  of  existing  student  assessment  in  our  schools. 
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Table  1 

Comparison  of  Assessment  Methodologies 


Old  Nets 


assess  student  achievement  after  3, 6,  and 
9  years  of  schooling 

assess  student  achievement  relative  to 
grades  3, 6,  and  9  curriculum 

assess  student  achievement  relative  to 
language  arts,  mathematics,  science,  and 
social  studies 

school  and  district  information  about 
student  achievement  at  four  points  in  the 
student's  career 

assessment  is  external  to  instruction  - 
marked  by  someone  other  than  the 
student's  teacher 

assessment  based  on  student  responses  on 
a  "paper  and  pencil"  measure 


individual  student  performance  on  four 
tests  analyzed  by  outcome  for  use  by 
teacher 


New  Nets 


assess  student  achievement  after  each  year  of 
schooling 

assess  student  achievement  relative  to  all  levels 
of  the  curriculum 

assess  student  achievement  relative  to  the  total 
curriculum 


school  and  district  information  about  student 
growth  in  achievement  on  an  annual  basis 


assessment  is  part  of  instruction  -  determined 
by  the  student's  teacher  -  subject  to  validation 
and  audit  at  the  district  level 

assessment  based  on  student  performance  of 
assigned  tasks  including  but  not  limited  to 
"paper  and  pencil"  measures 

individual  student  performance  on  tasks  used  by 
teacher,  student,  and  parent  for  diagnosis, 
planning,  and  celebration  of  success 


Measuring  Growth 

1 .  Field  Test  of  Prototype  Task  -  A  prototype  task  was  developed  and  field-tested 
with  year  one  students  (that  is,  students  in  their  first  year  of  schooling,  excluding 
kindergarten)  in  nine  elementary  schools  in  June  1990.  The  multiple  activity  task 
was  based  on  the  general  theme  of  "change"  and  required  the  students  to  participate 
in  an  activity  related  to  at  least  two  writing  activities,  drawing,  self-reflection,  and 
reading. 

Teachers  made  judgments  on  grade  level  of  achievement  based  on  the  initial  set 
of  achievement  criteria.  The  field  test  addressed  essential  learning  outcomes, 
indicators,  grade  level  of  achievement,  and  identification  of  the  sources  from 
which  teacher  judgments  were  made.  The  essential  learning  outcomes  and 
selected  indicators  are  shown  in  Table  2. 
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Table  2 


Learning  Outcomes  and  Selected  Indicators 


Learning  Outcome 

Selected  Indicators 

communication 

reading 

writing 

responsible  citizenship 

participating 

cooneratin? 

providing  service 

well-being 

making  choices 

exhibiting  nositive  life  habits 

knowledge 

applying 

calculating 

inquiry 

hypothesizing 

researching 

decision  making 

aesthetic  appreciation 

valuing 

expressing 

The  results  of  the  prototype  task  in  June  1990  were  analyzed  with  the  objective  of 
developing  achievement  criteria  and  a  more  refined  task  for  administration  in  the 
1990-91  school  year.  The  analysis  and  developmental  work  were  carried  out  by 
working  committees  of  teachers,  district  consultants,  and  curriculum  specialists. 

Inservice  training  was  provided  in  May  1991  to  teachers  of  year  one  students 
in  nine  schools  selected  to  pilot  the  more  comprehensive  measure  which  was 
administered  in  May  1991  to  year  one  students  (other  years  optional)  in  nine 
pilot  schools.  The  student  products  were  collected  for  analysis  and  central 
scoring  in  July  1991. 

In  1991-92,  trials  were  carried  out  with  tasks  developed  for  students  in  years  one 
to  twelve  in  40  demonstration  sites  was  well  as  cohort  participation  of  students  now 
in  year  two  in  the  nine  pilot  schools.  Analysis  of  the  results  of  these  measures  is 
continuing  in  1992-93,  in  preparation  for  administration  of  the  tasks  as  growth 
measures  in  April  1993. 
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2 .  Assessment  Task  Development  and  Process  Chronology 

1989-  90    •  one  task  developed  and  field-tested  in  nine  elementary  schools 

1990-  91     •  prototype  task  developed  and  field-tested  in  nine  elementary 

schools  at  the  year  one  level 

•  achievement  criteria  established  and  validated  by  teacher, 
consultant,  and  curriculum  specialist  teams 

1991-  92     •  14  tasks  developed  and  the  1989-91  tasks  revised 

•  15  tasks  piloted  in  35  schools  representing  "years  in  school" 
one  to  twelve  (see  Table  3) 

•  each  task  cross-referenced  to  essential  learning  outcomes  and 
relevant  course  of  study  outcomes  and  expectations 

•  inservice  training  and  consultant  facilitation  to  complete 
judgments  for  each  essential  learning  outcome  and  selected 
indicator  provided  to  teachers 

•  achievement  information  recorded  by  teachers 

•  over  2,000  students  participated  in  the  administration  of 
the  tasks 

1992-  93     ♦  analysis  of  the  teacher  j udgments  against  criteria  provides  initial 

verification  of  the  validity  of  the  assessment  instruments  and 
strategies  (work  in  progress) 
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Table  3 

Assessment  Tasks  and  Essential  Learning  Outcomes 


Number  Task  Name  Essential  Learning 

Outcomes  Demonstrated* 


1 

Patterns 

All  6  outcomes 

2 

Relationships 

All  6  outcomes 

3 

Adapting  to  the  Environment 

All  6  outcomes 

4 

Opposites 

All  6  outcomes 

5 

Traffic  Flow 

All  6  outcomes 

6 

Points  of  View 

Communication 

7 

Linking  the  Past  to  Today 

Communication 

8 

Heroes  and  Heroines 

Communication 

9 

Literary  Traditions 

Communication 

10 

A  Greek  Myth 

Communication 

11 

Limits  to  Population 

Inquiry 

12 

"Roto-copters" 

Inquiry 

13 

Exploration  of  Shapes 

Inquiry 

14 

Transportation 

Inquiry 

15 

All  Summer  In  A  Day 

Inquiry 

Essential  learning  outcomes:  1  communication,  2  responsible  citizenship,  3  well-being, 
4  knowledge,  5  inquiry,  6  aesthetic  appreciation. 
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3 .  Achievement  Results  of  Assessment  Tasks  Piloted  in  1991  -92 


ACHIEVEMENT  OF  YEAR  ONE  STUDENTS  (ASSESSMENT 
TASKS  -  1991  PILOT) 


Reading 
Writing 
Participating 
Cooperating 
Providing  Service 
Making  Choices 
Positive  Life  Habits 
Applying 
Calculating 
Researching 
Making  Decisions 
Valuing 
Expressing 


70% 

H — 


80% 
— I — 


M  Exceeds  Grade  One  (%)   ■  Grade  One  (%) 


□  Not  Yet  Grade  One  (%) 


Figure  2:  Graph  of  Year  One  Achievement  (Pilot  Assessment  Tasks) 


Table  4 

Achievement  Data  —  Year  One  Students  (Pilot  Assessment  Tasks) 


Essential  Learning 

Indicator 

Exceeds 

Grade 

Not  Yet 

N=  (valid  cases 

Outcome 

Grade  One 

One 

Grade  One 

out  of  422) 

(%) 

(%) 

(%) 

COMMUNICATION 

Reading 

7.5 

59.5 

33.0 

348 

Writing 

11.7 

53.5 

34.8 

368 

RESPONSIBLE 

Participating 

8.3 

68.9 

22.8 

360 

CITIZENSHIP 

Cooperating 

5.8 

70.5 

23.7 

359 

Providing 

Service 

4.3 

64.5 

31.2 

349 

WELL-BEING 

Making 

Choices 

14.4 

67.6 

18.0 

367 

Positive  Life 

Habits 

15.9 

61.0 

23.1 

364 

KNOWLEDGE 

Applying 

10.0 

65.1 

25.0 

371 

Calculating 

8.1 

54.5 

37.4 

369 

INQUIRY 

Researching 

7.8 

65.0 

27.2 

371 

Making 

Decisions 

14.3 

63.5 

22.2 

370 

AESTHETIC 

Valuing 

9.2 

59.2 

31.6 

370 

APPRECIATION 

Expressing 

15.1 

63.1 

21.8 

371 
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4 .  Correlation  of  Grading  Judgments  of  Analysts  and  Teachers  -  All  student  work 
from  the  tasks  was  judged  separately  by  the  student's  teacher  and  by  analysts  who 
had  had  major  involvement  in  the  establishment  of  scoring  criteria.  The  levels  of 
agreement  on  each  indicator  and  the  resulting  correlations  are  shown  in  Figure  3 
and  Table  5. 


Witing 
RartidpEting 
Cooperating 
Providing  Service 
O     rVbking  Choices 
q  Positive  Life  Habits 
§  Applying 
CafcUabng 
Researching 
Making  Decisions 
Vailing 


20 


AGREErVBMT  OF  ANALYST  AND  TEAOB*  SCORES 

PBCEMWGE  CF  CASES 
30  40  SO  GO  70 


J  1 

|  Scores  Agree 


§  Scores  Differ  by  1  Grade  □  Scores  Differ  by  >1  Grade 


Figure  3:  Agreement  of  Analyst  and  Teacher  Judgments 


Table  5 


Teacher  Judgments 


Scores  Differ  by 

Scores  Differ  by 

Indicator 

Scores  Agree 

1  Grade 

>1  Grade 

Correlation 

Reading 

66.4 

30.2 

3.4 

0.5 

Writing 

72.8 

26.3 

0,8 

0.7 

Participating 

74.1 

24.5 

0.8 

0.6 

Cooperating 

71.1 

27.3 

1.7 

0.5 

Providing  Service 

70.2 

27.0 

2.8 

0.5 

Making  Choices 

77.7 

21.2 

0.9 

0.6 

Positive  Life  Habits 

76.9 

21.4 

1.7 

0.6 

Applying 

78.2 

20.8 

1.0 

0.7 

Calculating 

74.8 

23.6 

1.6 

0.6 

Researching 

81.5 

18.4 

0.5 

0.7 

Making  Decisions 

81.1 

18.4 

0.5 

0.7 

Valuing 

79.6 

19.2 

1.2 

0.7 

Expressing 

79.0 

20.5 

0.5 

0.7 
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5.  Student  Growth  Results 


Outcome  Date  -  Yr.  in  School  n 


Reading: 


30  40  50  60    70  80   90  100 


Writing: 


1992- Yr.  2 
1991  -  Yr.  1 
1992- Yr.  2 
1991  -  Yr.  1 
Participating:  1992.  Yr.2 

1991  -  Yr.  1 
Cooperating:  1992_Yr  2 

1991  -  Yr.  1 
Providing  Service:  1992  _  Yr  2 

1991  -  Yr.  1 
1992- Yr.2 
1991  -  Yr.  1 
1992- Yr.2 
1991  -  Yr.  1 
1992 
1991 
1992 
1991 
1992 
1991 
1992 


Making  Choices: 
Positive  Life  Habits: 
Applying: 
Calculating: 

Researching: 
Making  Decisions: 


Valuing: 


Expressing: 


0  Grade  <1 


Level  of  Achievement 

Grade  1    H  Grade  2    □  Grade  3 


Figure  4:  Percentage  of  Students  at  Given  Grade  Levels  of  Achievement 
Nine  Pilot  Schools,  204  Students,  1991  and  1992 
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Table  6 

Percentage  of  Students  At  Given  Grade  Levels  of  Achievement 


Indicator 

Grade  <1 

Grade  1 

Grade  2 

Grade  3 

Reading:  1991  Yr.  1 

34.0 

48.0 

18.0 

0.0 

1992  Yr.  2 

2.0 

24.0 

54.0 

22.0 

Writing:  1991  Yr.l 

43.0 

42.0 

16.0 

0.0 

1992  Yr.2 

1.0 

26.0 

58.0 

16.0 

Participating:  1991  Yr.l 

25.0 

60.0 

16.0 

0.0 

1992  Yr.  2 

2.0 

11.0 

69.0 

19.0 

Cooperating:  1991  Yr.l 

25.0 

62.0 

13.0 

0.0 

1992  Yr.  2 

2.0 

11.0 

71.0 

17.0 

Providing  Service:  1991  Yr.l 

23.0 

64.0 

13.0 

0.0 

1992  Yr.  2 

1.0 

11.0 

71.0 

18.0 

Making  Choices:  1991  Yr.l 

22.0 

63.0 

15.0 

0.0 

1992  Yr.  2 

1.0 

8.0 

71.0 

20.0 

Positive  Life  Habits:  1991  Yr.l 

22.0 

62.0 

16.0 

0.0 

1992  Yr.  2 

1.0 

9.0 

71.0 

20.0 

Applying  1991:  Yr.l 

24.0 

61.0 

15.0 

0.0 

1992  Yr.  2 

1.0 

14.0 

70.0 

15.0 

Calculating  1991:  Yr.l 

28.0 

60.0 

11.0 

0.0 

1992  Yr.  2 

1.0 

14.0 

72.0 

14.0 

Researching:  1991  Yr.l 

27.0 

62.0 

11.0 

0.0 

1992  Yr.  2 

1.0 

14.0 

74.0 

12.0 

Making  Decisions:  1991  Yr.  1 

26.0 

60.0 

14.0 

0.0 

1992  Yr.  2 

1.0 

14.0 

73.0 

13.0 

Valuing  1991:  Yr.  1 

19.0 

66.0 

15.0 

0.0 

1992  Yr.  2 

0.0 

9.0 

71.0 

21.0 

Expressing:  1991  Yr.  1 

16.0 

63.0 

21.0 

0.0 

1992  Yr.  2 

1.0 

11.0 

68.0 

21.0 
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AVERAGE  GRADE  LEVEL  OF  ACHIEVEMENT 

0.0  1.0  2.0  3.0 

Reading: 

Writing: 
Participating: 
Cooperating: 
Providing  Service: 
Making  Choices 
Positive  Life  Habits: 
Applying: 
Calculating: 

Researching: 
Making  Decisions: 
Valuing: 

Expressing: 

Date  and  Year  In  School 
EZ3  1991  -  Year  1    ■  1992  -  Year  2* 

♦Note.  Year  two  data  cover  a  two-year  period 
Figure  5:  Growth  in  Essential  Learning  Outcomes,  Nine  Pilot  Schools,  204  Students 
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Table  7 

Growth  in  Essential  Outcomes  (Nine  Pilot  Schools,  204  Students) 


Average  Grade  Level  of 

Average  Grade  Level  of 

Achievement  of  Year  One 

Achievement  of  Year  Two 

Indicator 

Students 

Students 

1991 

1992 

Reading 

0.9 

2.0 

Writing 

0.7 

1.9 

Participating 

0.9 

2.1 

Cooperating 

0.9 

2.0 

Providing  Service 

0.9 

2.1 

Making  Choices 

0.9 

2.1 

Positive  Life  Habits 

0.9 

2.1 

Applying 

0.9 

2.0 

Calculating 

0.8 

2.0 

Researching: 

0.8 

2.0 

Making  Decisions 

0.9 

2.0 

Valuing 

1.0 

2.1 

Expressing 

1.1 

2.1 

6 .  Development  of  Achievement  Criteria  -  Achievement  criteria  for  selected 
indicators  of  communication  for  grades  1  to  12  were  developed  in  1990-91  by  a 
committee  of  ten  teachers  and  two  consultants.  These  indicators  were  revised  and 
refined  in  July  1991  for  grade  1,  and  during  1991-92  for  grade  2.  Preliminary 
criteria  for  selected  indicators  of  inquiry  for  grades  1  to  12  were  developed  during 
1991-92  and  will  be  revised  and  refined  during  1992-93.  Prelirriinary  criteria  for 
the  selected  indicators  of  the  other  four  essential  learning  outcomes  will  be 
developed  during  1992-93. 


7.  Cwricular  Relevance  -  Developmental  work  has  proceeded  on  the  topic  of 
relating  essential  learning  outcomes  and  selected  indicators  to  language  arts, 
mathematics,  science,  social  studies,  health,  and  art. 


8 .  Assessment  Task  Administration  1991 -92  -  To  maintain  reliability  of  procedures 
and  results,  teachers  at  demonstration  sites  were  provided  with  specific  task 
administration  routines  for  each  of  the  15  tasks.  These  guidelines  included  a 
statement  of  purpose,  a  process  model,  and  an  outline  of  steps  and  activities. 


9 .  District  Implications  -  Multiple  strategies  and  measures  for  student  assessment 
are  now  included  in  district  practices  in  Edmonton  Public  Schools.  Current  and 
proposed  practices  in  each  of  the  193  schools  are  guided  by  the  principles  for 
assessment,  evaluation,  and  communication  of  student  growth  (see  Rationale). 
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Portfolio  Design 

Initial  work  on  establishing  suitable  physical  designs  for  a  student  portfolio  was  an 
important  catalyst  for  the  examination  of  more  vital  issues  such  as  the  possible 
range  of  contents  in  a  portfolio,  the  implications  of  regulations  under  the  School  Act 
that  define  contents  of  a  student  record  and  the  access  to  that  record,  and  logistical 
problems  that  could  be  associated  with  full  utilization  of  a  portfolio  assessment 
system.  Models  of  suitable  portfolio  "containers"  were  produced  but  it  soon 
became  clear  that  the  most  suitable  physical  arrangements  for  managing  most  of  the 
potential  contents  of  a  student's  portfolio  were  those  developed  at  the  local  school 
and  classroom  level.  The  number  of  methods  used  is  likely  to  be  as  numerous  as 
the  number  of  teachers  employing  portfolio  assessment  procedures. 


Prototype  Computer  Technology 

1 .  Pen-based  Systems  -  The  two  competing  systems  on  the  market,  Microsoft 
Windows  for  Pen  Computing  and  Go  Corporation's  PenPoint,  were  evaluated  for 
possible  application  as  data  collection  systems  for  teacher  use  in  the  classroom. 
The  findings  were: 

•  both  systems  were  unstable 

•  hand-printed  character  recognition  was  inconsistent  and  unreliable 

•  the  devices  were  bulky  and  unsuitable  for  teacher  use  in  the  classroom  setting 

•  acquisition  costs  are  high 

Clearly,  the  pen-based  systems  are  still  using  an  immature  technology.  No  further 
evaluation  or  development  of  suitable  application  software  is  contemplated  until  the 
second  generation  of  pen-based  computing  devices  is  developed. 


2 .  Prototype  User  Screens  -  A  series  of  prototype  screens  has  been  developed 
using  the  standard  "Windows"  screen  format  and  Visual  Basic  as  the  development 
language.  Development  is  continuing  in  parallel  with  the  development  of 
assessment  tasks  and  related  reporting  procedures  in  the  design  project  Static 
screen  capture  images  of  the  various  interfaces  developed  to  date  are  shown  in 
Figures  6  to  10. 

The  flexibility  of  the  design  tool  is  such  that  suitable  screens  and  the  underlying 
application  for  the  capture  and  presentation  of  student  growth  information  can  now 
be  developed  very  quickly,  including  the  incorporation  of  scanned  image  and 
recorded  sound  when  appropriate. 
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EQI  TECHNICAL  PROTOTYPE:  CRITERIA 


II 


DEMOGRAPHIC    GROWTH    PROFILES    |Print|    |Help|  |Quit| 


OUTCOME 


INDICATOR 


GRADE 


|Aesthic  Appreciation 


Expressing 


TASK: 


SUtti&tiis  exftfess  twlinsjz  and  weaning  Ihrough  painting.  &amng,  and  miiinsj.  Pi(ssim-Ax  ftw  ctrmlyxiz 
consist  of  painting,  liu-winy.  student  miiinq.,  teaches  ootcs.  and  student  {ejections  (vwilten  by 
student  or  scribed  by  teacher  frcra  student  dictation]. 


Work  shows  that  students: 

use  a  variety  of  colours  or  media  in  drawing/painting 

express  knowledge,  feelings,  and  ideas  through  (art)  work  (eg.,  draw  and  colour  with  pattern, 
illustrate  a  story). 

give  attention  to  detail  in  colour  and  drawing 


attempt  appropriate  scale  and  perspective  in  drawings 


jAestfoje  Appreciation:  txpressirtg:  1  \  Ur<S\ 


Student  Notes    Teacher  Notes 
Picture    Writing    Painting  Drawing 


■•  ■'-"''''y'Ktfi'i"- 


F/gwre  6:  A  Screen  to  Display  Criteria  and  a  Scanned  Image  of  Student  Work 


i&Ql  TECHNICAL I^OTOTYRE:  GfW3HICAL  COMPARISONS 


DEMOGRAPHIC    GROWTH    CRITERIA    PEOFILES    |Print|    |Help|  |Quit| 


Period 


mmm 


Student 


Grade 


Class 


School 


Area 


District 


Period 


SCHOOL  AVERAGE 


:'::::::;::S  ::::S:S     :  •'  - '.' ■  •:  :::: 


w 


M  U  U 


3  3  \ 


Student 

1 


Grade 


■ 


Class 

 TS 


School 


Area 


|j|    Communication  Responsible  Weil-Being    Knowledge  Inquiry       Aesthic  1 

Citizenship  Annrpriahnn 


Appreciation 


Rg4  I  ACHIEVEMENT]  j  j 

School  Scheduled  Task  District  Scheduled  Task 


District 


Figure  7:  A  Screen  to  Display  Graphical  Comparisons  of  Student, 
Class,  School,  District,  or  Date  Level 
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CO  I  TLCMMtCAL  PROTOTYPE:  &ROWTH  L)Y  SCI  IGDL  YEAR 


DEMOGRAPHIC     GROWTH    CRITERIA    PROFILES    |PRINT|    |HELP|  |QUIT| 


Student 


School 


m 


' Legend 
Hill  1990 

nn  1991 

mm  1992 


3  YEAR  HISTORY  STARING  FROM:  [1990 


Communicaton  Knowledge     Inquiry     Citizenship   Well-Being  Aesthetic 

Appreciation 


Figure  8:  A  Screen  to  Display  Student  Growth  by  School  Year 


mt mZWiiCM  «WQTYPf£:  GROWTH  BY  YEAR  \H  SCHOOL 


DEMOGRAPHIC    GROWTH    CRITERIA    PROFILES    |Print|    |Help|    |  Quit  | 


Student 


Johnny  B.  Good 


Class 


Year  In  School 


1    2   3   4   5  6 


2    3    4    5  G 


School 


2  3  4  5   G  7 


8  9 


1  2345G789 


Figure  9:  A  Screen  to  Display  Student  Growth  by  Year  in  School 
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mi  TECHNICAL  PROTOTYPE:  PROFILE 


DEMOGRAPHIC    GROWTH    CRITERIA    |Grjd|    |Printj    |Help|  |Quit| 


Essential  Learning 
Outcome 


Communication 


Student 


Year  In  School 


1 
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School 


1 
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YEAR  IN  SCHOOL 


Figure  10:  A  Screen  to  Display  Profiles  by  Student,  Class,  School,  Area,  or 
District  (Based  on  Range  of  Achievement  by  Year  in  School) 


3 .  Requirements  for  Wide  Area  Network  Upgrading  -  The  technical  requirements 
for  an  upgraded  wide  area  network  have  been  identified.  These  address  the  need  to 
accommodate  high  data  traffic  flow,  image  transfer,  distributed  processing  of  data 
where  appropriate,  and  internal  school  and  central  connections  to  local  area  networks. 

The  recent  developments  in  data  communications  standards  and  implementation  of 
fibre  optic  systems  have  made  the  technical  concept  feasible.  The  wide  area  network, 
which  would  interconnect  all  schools  and  central  operating  units  independent  of 
computer  type,  will  meet  the  data  communication  needs  and  storage  demands  of 
the  district's  monitoring  functions.  Issues  that  continue  to  be  addressed  include: 

•  affordability 

•  cooperative  line  charge  arrangements  with  the  carriers 

•  continuing  instability  of  local  area  networks 

•  security  of  data 

•  the  role  of  "multimedia"  in  the  assessment  process  and  the  technical  challenge  of 
handling  heavy  data  traffic  on  a  wide  area  network 
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4 .  Computer  Assisted  Student  Progress  Reporting  -  A  flexible  and  stable 
application  has  been  written  which  allows  teachers  to  key-enter  their  own 
commentary  for  each  student  and  produce  a  centrally  printed  progress  report, 
via  local  area  network  to  the  school  office.  The  report  includes  the  student's 
picture,  comments,  and  "marks".  The  system  is  in  use  in  three  schools  and 
incorporates  many  design  features  suggested  by  classroom  teachers. 

There  is  a  clear  connection  between  this  developmental  work  and  the  work  related 
to  user  screens  for  presentation  and  capture  of  student  growth  information.  The 
current  progress  reporting  system  is  designed  for  the  Macintosh  computer  but  it  is 
planned  that  a  DOS/Windows  version  will  shortly  be  available  for  use  by  a  wider 
range  of  schools.  The  system  is  compatible  with  wide  area  network  standards.  It 
is  anticipated  that  the  two  initiatives,  progress  reporting  and  student  growth  user 
screens,  will  be  merged. 


Conclusions 

Although  there  remain  many  unanswered  questions  and  much  work  still  has  to  be 
done,  the  findings  from  the  first  three  years  of  the  project  are  encouraging  and  give 
promise  for  the  future.  Teacher  and  student  comments  indicate  that  the  tasks  and 
results  can  be  useful  in  enhancing  the  reporting  of  student  achievement  and  growth. 
The  data  derived  from  the  teacher  evaluations  can  be  used  by  teachers  as  part  of 
their  assessment  of  the  individual  student's  achievement  and  growth  and  can  be 
aggregated  at  the  school  and  district  level  to  provide  useful  accountability 
information.  At  the  same  time,  portfolios  of  student  work  can  be  used  to  discuss 
and  validate  a  student's  individual  progress  during  conferences  among  teachers, 
parents,  and  students. 

The  feasibility  of  developing  easy  to  use  computer  applications  for  the  capture, 
display,  and  communication  of  students  and  growth  has  been  established. 


Follow-up 

•  Revision  and  design  of  tasks  for  students  in  years  one  to  twelve 

•  Preparation  of  teachers  for  task  administration 

•  Administration  of  revised  and  developed  tasks  in  spring  1993 

•  Refinement,  development,  and  application  of  achievement  criteria 

•  Judgment  of  student  achievement  in  the  demonstration  schools 

•  Collection  and  reporting  of  growth  information 

•  Development  of  paper  and  computer  formats  for  display  of  growth  data 
at  all  levels 

•  Preparation  of  a  staff  development  plan 

•  Attempt  to  resolve  identified  issues 
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Educational  Quality  Indicators  in 
Art  and  Mathematics 


Calgary  School  District  No.  19  and  Calgary  RCSSD  No.  1 


The  Calgary  Educational  Quality  Indicators  (EQI)  Project  was  a  three-year  action 
research  project  (1989  -  1992)  funded  by  Alberta  Education  to  develop  indicators 
of  quality  student  performance  in  school  art  composition  and  mathematical  problem 
solving  at  the  Calgary  Board  of  Education  (CBE)  and  the  Calgary  Catholic  Board  of 
Education  (CCBE). 


Purposes 

The  purposes  of  the  EQI  Project  were  to: 

•  identify  indicators  of  quality  work  which  can  be  applied  in  the  assessment  of 
student  achievement  in  areas  of  art  and  mathematics 

•  identify  conditions  that  appear  to  enhance  or  limit  the  achievement  of  quality 

•  develop  strategies  for  collecting  quality  student  work  which  include  recorded 
interviews  with  teachers  and  students 

•  examine  quality  work  with  emphasis  on  alternative  assessment  strategies  to 
paper-and-pencil  testing 

The  project  was  part  of  a  larger,  province- wide  initiative  sponsored  by  Alberta 
Education  to  build  sets  or  clusters  of  indicators  to  use  in  a  broader  approach  to 
assessment. 


Rationale 

The  Calgary  project  focused  on  assessing  performance  in  the  areas  of  art  and 
mathematics.  These  subject  areas  were  chosen  because  of  preliminary  work  which 
had  already  been  done  at  the  CCBE  in  the  area  of  art  assessment  and  because  of 
mathematics  expertise  at  the  CBE.  Further,  it  was  reasoned  that  the  two  areas  could 
allow  a  unique  exploration  of  the  assessment  of  student  product  and  process.  The 
art  project  could  examine  the  product  of  the  learning,  as  much  of  art  education  had 
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stressed  process  in  the  past,  and  the  math  project  could  examine  the  process  of 
student  thinking,  as  most  math  education  had  emphasized  the  product,  or  answer, 
in  the  past. 


Design 

The  project  design  incorporated  three  phases:  the  development,  field  testing  and 
pilot  testing  of  quality  indicators.  Committees  of  teachers  and  subject  specialists 
reviewed  student  work,  identified  characteristics  of  quality  work,  considered 
alternative  approaches  for  recording  assessment  information,  and  proposed 
activities  and  materials  for  use  in  classrooms.  Classroom  teachers  tried  out  a 
variety  of  materials  and  approaches  with  their  students,  materials  were  revised 
and  then  were  piloted  with  other  teachers. 

The  project  had  a  full-time  designated  director  who  was  located  at  the  Calgary 
Board  of  Education.  A  steering  committee,  comprised  of  school  system 
superintendents,  subject  area  specialists,  and  representatives  from  Alberta 
Education  provided  coordination  and  leadership.  A  community  advisory 
committee,  comprised  of  university  professors,  leaders  in  the  fields  of  art  and 
mathematics,  parent  representatives,  representatives  of  the  Alberta  Teachers' 
Association,  and  members  of  the  business  community  provided  a  sounding  board 
to  test  project  ideas.  Finally,  two  subject  area  advisory  committees,  with  specialists 
from  both  boards,  provided  "hands-on"  decision  making  for  the  action  research  in 
each  area. 

Tables  1  and  2  summarize  the  involvement  of  staff,  the  tasks  performed,  and  the 
results  of  the  three  years  of  the  project. 


Findings 

Art 

The  art  component  of  the  project  culminated  in  the  development  of  the  Educational 
Quality  Indicators  Handbook.  The  handbook  includes  color  reproductions  of 
quality  samples  of  students'  art  from  grades  2  and  5,  and  from  junior  and  senior 
high  schools.  Teacher  expectations  for  each  lesson  and  student  comments 
assessing  their  own  work  are  also  included.  Three  quality  indicators  emerged  from 
the  project  as  observable  characteristics  to  be  found  in  students'  work.  They  were 
relationships,  handling  and  meaning.  This  work  is  linked  to  the  Alberta  Education 
program  of  studies  in  art.  Student  and  teacher  journals  were  used  in  the  pilot  test  to 
assist  in  reflective  practice  on  the  part  of  both  teachers  and  students. 

Project  findings  for  the  art  component  included  the  following: 

•  Quality  assessment  practice  supports  improved  art  making. 

•  Attending  to  written  and/or  spoken  student  responses,  reflections  and  thoughts 
iUuminates  the  art  work. 

•  Quality  art  assessment  is  a  shared  experience  between  students  and  teachers. 
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•  Effective  art  assessment  depends  on  common  vocabulary  shared  by  teachers 
and  students. 

•  The  collection  of  quality  art  exemplars  emphasizes  diversity  as  an  attribute  of  a 
quality  art  program. 

•  Expert  professional  judgment  benefits  from  collegial  work. 

•  Quality  assessment  takes  time. 

•  EQI  handbook  exemplars  are  a  way  to  share  expertise  and  to  develop 
connoisseurship. 

Mathematics 

The  math  component  explored  the  concept  of  holistic  scoring  and  developed  some 
response  criteria.  These  included  preliminary  response,  partial  response,  complete 
response  and  elegant  response.  A  series  of  students'  mathematical  dispositions 
were  identified,  including  motivated,  creative,  confident,  strategic  approach  and 
strategic  process.  Conditions  for  the  math  learning  environment  were  also 
considered. 

Project  findings  for  the  mathematics  component  included  the  following: 

•  Teachers  report  positive  changes  to  their  teaching  and  assessment  practices  as  a 
result  of  participating  in  EQI-Math. 

•  Teachers  want  to  meet  to  discuss  teaching  and  assessment.  The  experience  of 
working  closely  with  colleagues  was  very  positively  received. 

•  Useful  materials  and  practices  developed  in  the  project  include  student 
dispositions,  holistic  scoring  and  the  exploration  of  diverse  student  responses 
and  quality  solutions  to  mathematical  problems. 

•  The  use  of  anecdotal  records,  student  attitude  reports  and  generalizations  about 
classroom  conditions  are  not  seen  as  practical,  due  to  time  and  energy 
constraints. 

•  Students  seem  to  be  more  positive  about  problem  solving,  take  more  risks,  and 
demonstrate  more  confidence  as  a  result  of  the  EQI  project. 

•  Systematic  methods  for  observing  students  closely  help  teachers  with  teaching 
decisions  and  with  assessment  and  evaluation  tasks. 

•  Students  respond  to  invitations  to  express  themselves  more  fully  and  to  think 
about  their  thinking. 
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Common  Findings 

As  the  EQI  project  evolved  over  the  course  of  three  years,  commonalities  emerged 
about  the  making  of  art  and  the  solving  of  mathematical  problems.  They  include: 

•  Complexity  of  thinking  processes  are  common  to  both  art  and  math  problem 
solving. 

•  Art  and  math  present  assessment  challenges  to  teachers. 

•  Quality  assessment  practice  in  art  and  mathematics  supports  teacher  decision 
making  about  teaching  and  assessment. 

•  Attending  to  written  and  spoken  student  responses,  reflections  and  thoughts 
illuminates  the  art  work  or  the  problem  solution. 

•  Quality  assessment  is  a  shared  experience  between  teachers  and  students  which 
is  encouraged  by  shared  exemplars  and  a  shared  vocabulary. 

•  Qualitative  assessment  takes  time  for  both  teachers  and  students. 

•  Exemplars  are  a  way  to  share  expertise,  develop  connoisseurship,  establish 
common  expectations  for  student  outcomes  and  celebrate  diversity  of  student 
response. 


Other  Observations  and  Conclusions 

Learning  theory  provides  us  with  the  view  of  the  learner  as  an  active  meaning 
maker,  constantly  making  decisions  and  adapting  new  experiences  in  the  making 
of  new  knowledge.  Corresponding  to  this  view  of  the  learner,  the  EQI  project 
assessment  procedures  assisted  in  illuminating  the  learner  outcomes,  celebrated  and 
encouraged  diversity  of  outcomes,  and  helped  both  teachers  and  learners  share  in 
the  assessment  process.  As  a  result  of  this  sharing,  researchers  observed  the 
following: 

•  Many  educators  are  attempting  to  implement  new  assessment  techniques 
through  the  use  of  student  portfolios,  performances  and  extended  projects.  The 
materials  from  EQI  Calgary  provide  some  foundational  understandings  and 
methods  that  would  give  these  new  forms  of  assessment  increased  validity. 

•  Informed  professional  judgment  develops  when  teachers  have  opportunities 
to  work  with  colleagues  and  share  new  strategies  and  insights. 

•  The  EQI-Math  teacher  researchers  and  pilot  teachers  report  that  discussing 
student  responses  with  other  teachers  is  an  important  benefit  of  participation 
in  this  project. 

•  A  logical  extension  of  the  EQI  practice  of  assessment  is  its  use  in  formal 
evaluation  and  reporting  practices. 
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♦    The  findings  and  conclusions  from  this  project  suggest  that  the  processes  used 
for  developing  new  approaches  to  assessment  in  art  and  mathematics  may  be 
valuable  for  other  subject  areas  as  well. 


Recommendations 

At  the  conclusion  of  the  Calgary  EQI  Project  participants  at  all  levels  assisted  in  the 
preparation  of  recommendations  for  future  development  of  the  project  ideas  and 
materials.  The  participants  recommended  that: 

•  the  findings  from  this  study  be  incorporated  into  evaluation  and  reporting 
practices 

•  the  EQI- Art  Handbook  and  the  Problem  Solving  Profiles  be  published  and 
made  available  to  schools  within  the  CBE,  the  CCBE,  and  to  other  school 
systems  on  request 

•  teacher  education  include  a  variety  of  assessment  skills  in  course  work 

•  teachers  be  encouraged  to  develop  professional  collections  of  quality  student 
exemplars  in  art  and  in  mathematics 

•  teachers  be  provided  with  opportunities  to  work  on  collegial  assessment  of 
student  work 

•  school  jurisdictions  be  invited  to  expand  the  EQI  project  process  to  include 
other  disciplines 

In  conclusion,  this  project  reaffirms  the  importance  of  thoughtful  reflection  on  the 
part  of  the  students  and  teachers  in  the  assessment  of  student  work.  It  is  also  clear 
that  innovation  requires  system  support,  guidance  and  time.  When  teachers  and 
students  are  informed  by  experts,  connoisseurs  and  the  literature,  they  have  the 
knowledge  and  skill  to  engage  in  meaningful  assessment. 
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Academic  Outcomes  and  Behavior 


-  Identifying  Quality  Indicators 


Brooks  School  District  No.  2092 


The  purpose  of  the  Brooks  EQI  Project  was  to  study  student  behavior  and  academic 
achievement  We  identified  a  set  of  quality  indicators  of  responsible  student 
behavior  in  the  schools.  The  community  was  involved  in  promoting  mutually 
supportive  efforts  in  maintaining  and  improving  student  behavior.  As  an  outcome 
of  our  project,  we  developed  a  set  of  measures  and  a  set  of  community  beliefs 
regarding  the  behavior  of  our  youth. 


Context 

The  Brooks  School  District  is  located  in  the  town  of  Brooks  which  has  a  population 
of  9,500.  The  town  is  surrounded  by  257,000  acres  of  irrigated  farmland  and  over 
a  million  acres  of  rangeland  used  for  livestock  grazing.  The  lands  surrounding 
Brooks  support  an  active  oil  industry. 

The  Brooks  School  District's  six  schools  are  located  within  an  eight-block  radius  in 
the  town  of  Brooks  in  southeastern  Alberta.  The  community  is  stable  and  there  is 
much  pride  in  the  school  district.  In  September  1991,  there  were  2,372  students, 
127  teachers  and  46  support  staff. 


Purposes 

The  purposes  of  the  Educational  Quality  Indicators  (EQI)  Project  in  the  Brooks 
School  District  were: 

1 .  to  study  the  validity  of  the  premise:  quality  education  in  Brooks  is  perceived  to 
be  the  result  of  the  "right"  blend  of  attention  to  academic  success  and  attention 
to  development  of  "character"  within  the  framework  of  a  firm,  but  supportive  set 
of  behavioral  expectations  and  policies;  and 

2.  as  an  outcome  of  addressing  the  premise  we  expect  to  define  a  valid  set  of 
quality  indicators. 
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The  expected  long-term  results  are: 

•  higher  satisfaction  and  confidence  among  students,  graduates,  and  employees 
with  respect  to  the  educational  product  as  measured  by  staff  questionnaires, 
administrator  interviews,  student  homework  completion  rate,  vandalism  rate, 
discipline  rate,  former  student  survey,  and  telephone  interviews; 

•  a  higher  than  normal  percentage  of  counseling/discipline  related  cases  (for 
example,  our  denial  of  student  privileges,  suspension  and  expulsion  rates  may 
be  higher  due  to  our  firm  stand  on  acceptable  behavior); 

•  a  lower  dropout  rate; 

•  maintenance  of  high  academic  achievement  test  results,  diploma  exam  results, 
participation  rates  and  Rutherford  Scholarships; 

•  wider  acceptance  of  the  quality  indicators  developed  by  stakeholders  in  Brooks. 

The  literature  suggests  that  the  result  of  the  "right"  blend  of  attention  to  academic 
success  and  development  of  "character"  depends  on  the  student  and  circumstance. 
The  values  taught  at  school  as  well  as  in  the  home  are  important.  It  is  clear  that 
both  ingredients  must  exist  in  order  for  quality  education  to  occur.  The  Brooks  EQI 
Project  concentrated  on  desirable  student  behavior  and  outcomes  on  a  school  district 
basis. 


Design 

The  Brooks  EQI  Project  spanned  three  years.  The  first  stage  of  the  project,  1989- 
1990,  identified  a  set  of  quality  indicators.  Interviews  were  conducted  with 
administrators  and  trustees,  and  teachers  provided  their  perceptions  in  writing; 
these  formed  the  basis  for  a  questionnaire  that  was  completed  by  the  school  district 
personnel.  Information  from  these  sources  was  used  to  identify  the  quality 
indicators. 

The  second  stage  of  the  project,  1990-1991,  involved  collecting  data  in  the  following 
areas:  a  survey  by  former  students;  information  on  counseling/  discipline;  homework 
completion  rates;  interviews  on  a  very  small  sample  of  former  students  who  had  been 
expelled  or  received  a  Rutherford  Scholarship.  Data  were  subsequently  collected  on 
school  vandalism.  Achievement  test  results  were  analyzed  for  students  in  grades  3, 6, 
9  and  diploma  exam  results  for  grade  12.  An  action  plan  was  devised  with 
administrators  to  address  the  information  collected. 

The  third  stage  of  the  project,  1991-1992,  implemented  the  action  plan.  The 
community  was  involved  in  completing  a  profile,  and  developing  a  set  of  beliefs 
and  a  tentative  action  plan  to  follow  up  on  perceived  needs. 
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Responsible  Student  Behavior 


One  indicator  of  a  good  school  district  is  high  expectations  of  student  behavior. 
The  Brooks  School  District  chose  to  study  outcomes  of  student  behavior  in  terms  of 
homework  completion,  and  lack  of  vandalism.  We  also  collected  student  opinion  on 
standards  of  conduct  in  our  schools.  Administrators  who  actively  counsel  students 
and  a  supportive  school  board  collaborate  to  promote  positive  student  behavior. 

Indicators  of  responsible  student  behavior  included  high  expectations,  mutual 
respect  and  recognition,  pride  in  self,  others,  home,  school  and  community, 
courteous,  supportive  caring  and  sharing  attributes,  and  mutual  trust  and  open 
communication. 

The  following  model  illustrates  the  indicators  of  responsible  student  behavior  in  the 
community. 


Figure  1:  EQI  Model  of  Indicators  of  Responsible  Student  Behavior 


•    High  expectations  -  The  staff  and  trustees  of  the  Brooks  School  District  believe 
that  adults  must  have  high  expectations  of  student  behavior.  Students  must 
internalize  the  expectations  and  in  turn  demonstrate  self-discipline.  In  order  to 
promote  self-discipline,  adults  must  consistendy  help  children  to  see  themselves 
acting  responsibly  rather  than  constantly  having  to  enforce  positive  behaviors. 
All  Brooks  staff  and  trustees  demonstrate  a  commitment  to  developing 
responsible  behavior. 
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•  Mutual  respect  and  recognition  -  Encouragement  to  celebrate  positive  and 
responsible  behavior  is  provided  through  counseling  and  coaching.  Respect  for 
the  rights  of  others  and  a  sense  of  tolerance  and  understanding  are  promoted. 
Adults,  especially  teachers,  model  mutual  respect  and  recognition. 

•  Pride  in  self,  others,  home,  school  and  community  -  The  personal  appearance  of 
staff  and  students  as  well  as  the  physical  appearance  of  school  buildings  show  a 
sense  of  pride.  The  teachers  actively  teach  pride  through,  for  example,  the  care  of 
desks.  Pride  is  encouraged  through  the  celebration  of  student  accomplishments, 
such  as  honor  roll  recognition,  scholarships,  and  so  forth. 

•  Courteous,  supportive,  caring  and  sharing  attributes  -  Visitors  comment  on  the 
friendliness  and  courtesy  shown  to  them  by  students.  Cooperative  learning 
encourages  responsible  student  behavior  in  the  classroom.  Integration  of 
special  needs  students  is  encouraged. 

•  Mutual  trust  and  open  communication  -  Students  take  appropriate  risks  and 
speak  their  mind,  joke  with  adults,  question  policies,  and  so  forth.  Schools 
trust  students  to  live  up  to  expectations.  Trust  and  open  communication  are 
modeled  by  teachers  and  administrators.  There  is  a  positive  climate  within 
the  Brooks  School  District. 


Findings 

In  the  first  year  of  the  project,  data  collection  included  staff  perceptions,  a  district 
questionnaire  and  interviews  with  administrators.  This  resulted  in  a  model  of 
responsible  student  behavior.  During  the  second  year,  data  collection  expanded 
to  include  the  following  areas:  homework  completion,  vandalism,  counseling/ 
discipline,  former  student  survey,  telephone  interviews,  achievement  test  and 
diploma  exam  results,  participation  rates  and  Rutherford  Scholarships.  They 
measure  the  indicators  of  responsible  student  behavior  and  high  expectations. 
The  results  follow. 

1 .  Homework  Completion  -  In  the  fall  of  1990,  students  from  grades  4  to  12  were 
surveyed  to  determine  the  amount  of  homework  completed  in  the  Brooks 
School  District  on  a  Homework  Completion  Rate  form.  The  data  were  collected 
by  teachers  over  a  two-week  period.  Homework  was  defined  as  "any  student 
work  (other  than  studying  or  reading)  that  requires  time  to  complete  at  home". 
The  rate  of  return  by  the  elementary  teachers  was  100%,  the  overall  rate  of 
return  was  70.6%.  Teachers  of  subject  areas  without  homework  such  as  band 
and  industrial  arts  were  not  involved  in  the  task.  Homework  completion  rates 
for  the  five  participating  schools  varied  from  78.4%  to  91.1%.  The  overall 
Brooks  School  District  homework  completion  rate  was  84.9%. 

2 .  Vandalism  -  Vandalism  rates  were  collected  from  September  1990  to  December 
1990.  Vandalism  was  defined  as  any  action  which  results  in  maintenance  being 
involved  in  cleanup/repair,  for  example,  broken  window,  graffiti,  etc.  There 
was  no  vandalism  recorded  during  that  four-month  period. 
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3 .  Counseling/Discipline  -  In  the  fall  of  1990,  administrators  collected  information 
regarding  counseling/discipline  in  the  Brooks  School  District  The  objective 
was  to  attempt  to  determine  the  number  of  students  disciplined  by  an 
administrator,  and  the  effect  of  the  discipline  action.  Discipline  was  defined  as 
any  adininistrator  contact  with  students  at  teacher  request.  The  schools  each 
chose  a  two- week  period  in  which  to  collect  the  nature  of  the  problem,  action 
taken,  time  involved,  and  if  possible,  results  of  the  action. 

In  total  103  students  were  dealt  with  during  the  two- week  period:  36  cases 
related  to  attendance,  14  to  safety,  13  work  habits,  18  attitude,  14  fighting,  3 
profanity,  4  smoking  and  1  drinking  related  problem.  Action  was  taken  in  100 
cases:  28  follow-up  (included  detention,  student  contract,  loss  of  privileges, 
work  packages  and  in- school  suspension),  47  parent  contacts,  5  agency 
contacts,  18  school  district  contacts  (including  suspensions)  and  2  school  board 
contacts  (including  expulsions).  Thirty-three  cases  were  elementary  and  64 
were  secondary  students  (6  were  unidentified  as  they  were  from  a  school 
having  both  elementary  and  secondary  students). 

4.  Former  Student  Survey  -  In  1991,  former  students  were  sent  a  40-item  questionnaire 
asking  their  opinions  about  their  schooling  in  the  Brooks  School  District  The 
questions  were  organized  into  12  categories:  student  behavior,  discipline/ 
attendance,  homework,  school  cleanliness,  satisfaction,  student  problems, 
extracurricular  activities,  teacher/administrator,  fairness/recognition,  individual 
attention,  parent  involvement,  and  school  involvement.  Using  a  4-point  scale, 
respondents  were  asked  to  indicate  agreement  or  disagreement  with  the  statements. 
The  Former  Student  Survey  was  sent  to  749  former  students;  175  (23%)  responded. 
Almost  half  (49%)  of  the  respondents  had  spent  their  entire  educational  career  in 
Brooks,  while  25%  had  spent  high  school  only.  The  students  who  indicated  post- 
secondary  school  plans  included:  a  majority  (71%)  planned  to  attend  technical 
school,  college  or  university,  23%  planned  to  work,  2%  planned  to  attend  another 
high  school  and  1%  planned  to  return  to  Brooks  Composite  High  School  to 
graduate.  Respondents  reported  that  17%  received  an  advanced  diploma  with 
excellence,  42%  received  an  advanced  diploma,  35%  received  a  general  diploma, 
while  5%  indicated  choosing  not  to  complete  high  school. 

Results  indicated  that  students  agreed  most  that:  they  learned  to  solve  their  own 
problems;  teachers  were  willing  to  give  students  individual  help  outside  of  class 
time;  the  school(s)  maintained  a  satisfactory  appearance;  students  were  expected 
to  behave  in  an  acceptable  manner,  students  knew  what  was  expected  of  them  in 
school;  and  there  was  general  satisfaction  with  the  school(s),  their  teachers  and  the 
education  received  from  the  Brooks  School  District  Students  disagreed  the  most 
with  the  following:  student  respect  for  the  property  of  others;  other  students' 
ability  to  understand  their  problems;  students  receiving  satisfactory  recognition  for 
responsible  behavior  (e.g.,  telling  the  truth,  keeping  notebooks  and  locker  clean, 
playing  well  with  others,  etc.);  the  emphasis  placed  on  the  social  development  of 
students;  the  help  to  students  to  make  the  transition  out  of  grade  12;  the  school's 
guidance  and  counseling  services  were  satisfactory;  the  job  of  helping  students 
understand  their  moral  and  ethical  responsibilities;  recognition  for  extracurricular 
accomplishments,  and  additional  time  being  spent  on  homework  and  home  study 
per  school  night. 
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5 .  Student  Interviews  -  Telephone  interviews  were  conducted  with  students  who 
had  left  school  at  least  three  years  ago.  Students  chosen  for  interviews  were 
either  Rutherford  Scholarship  students  or  students  who  had  been  expelled.  The 
purpose  of  the  interview  was  to  determine  former  students'  perceptions  about 
student  behavior  within  the  Brooks  schools.  Thirty  phone  calls  were  attempted 
to  expelled  students.  Many  of  the  students  had  changed  residences,  weren't 
available  or  didn't  call  back  as  requested.  Three  requests  were  refused  and  three 
interviews  were  completed.  Thirteen  calls  were  attempted  to  scholarship 
recipients.  Many  of  these  students  were  at  university.  Zero  requests  were 
refused  and  four  interviews  were  completed. 

Of  the  seven  interviews,  two  students  questioned  school  board  policies  on 
attendance  and  required  student  behavior.  Two  students  felt  that  the  policies 
had  an  effect  on  student  behavior.  All  the  students  felt  that  policies  had  some 
effect  on  all  students.  Many  felt  that  the  home  had  a  stronger  influence  than 
school  district  policies.  Students  indicated  that  parents  supported  the  school 
policies  in  all  Rutherford  student  cases,  and  one  expelled  student  case.  Five 
of  the  students  were  planning  on  attending  college  or  university  and  two 
respondents  were  working. 

6.  Achievement  -  Provincial  achievement  test  results  in  grades  3, 6  and  9  and 
diploma  exam  results  in  grade  12  for  1989-90  were  reviewed.  Tables  1  through 
3  present  the  results. 


Table  1 

Percentage  of  Grade  3,  6  and  9  Students  Achieving  Standards 
on  the  Total  Provincial  Achievement  Tests,  June  1990 


Achievement  Acceptable  Standard*     Standard  of  Excellence 

Tests  District      Province  District  Province 


Grade  3  Mathematics 

97.7 

91.8 

34.9 

26.9 

Grade  6  Science 

85.6 

82.5 

24.8 

24.6 

Grade  9  Language  Arts 

82.9 

82.9 

8.5 

10.8 

*  Includes  standard  of  excellence. 
Source:  Alberta  Education 
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Table  2 


Percentage  of  Grade  12  Students  Achieving  Standards  on  the 
Diploma  Exams  (Final  Blended  Mark),  January  1990 


Diploma 

Acceptable  Standard* 

Standard  of  Excellence 

Exams 

District 

Province 

District 

Province 

English  30 

95.7 

95.3 

8.7 

12.2 

English  33 

100.0 

94.0 

0.0 

3.8 

Social  Studies  30 

88.6 

90.3 

9.1 

14.1 

Mathematics  30 

83.7 

86.8 

23.3 

22.3 

Biology  30 

97.7 

88.7 

23.3 

18.3 

Chemistry  30 

94.2 

89.1 

21.2 

20.2 

Physics  30 

78.6 

88.8 

21.4 

22.6 

*  Includes  standard  of  excellence. 
Source:  Alberta  Education 


Table  3 

Percentage  of  Grade  12  Students  Achieving  Standards  on  the 
Diploma  Exams  (Final  Blended  Mark),  June  1990 


Diploma 

Exams 

Acceptable  Standard* 

District  Province 

Standard  of  Excellence 

District  Province 

English  30 

94.3 

94.9 

5.7 

12.8 

English  33 

90.0 

90.8 

0.0 

2.3 

Social  Studies  30 

96.1 

89.1 

11.8 

15.4 

Mathematics  30 

82.9 

82.8 

14.6 

18.0 

Biology  30 

96.2 

84.9 

24.5 

19.7 

Chemistry  30 

85.7 

86.1 

19.0 

21.3 

Physics  30 

93.8 

90.7 

25.0 

27.6 

*    Includes  standard  of  excellence. 
Source:  Alberta  Education 
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7 .  Participation  -  Table  4  presents  the  percentage  of  students  who  wrote  provincial 
achievement  tests  and  diploma  exams.  The  grade  3, 6,  and  9  participation  rates 
are  from  documentation  provided  by  Alberta  Education.  The  participation  rate 
for  the  diploma  exams  was  based  on  the  number  of  students  who  wrote  the  tests 
compared  to  the  September  30, 1989  enrollment.  Brooks'  participation  in  these 
tests  was  higher  than  overall  provincial  participation  in  all  subjects  tested  except 
grade  9  language  arts. 

8 .  Rutherford  Scholarship  -  The  total  number  of  1989-90  grade  12  students 
receiving  a  portion  or  total  scholarship  was  27;  15  student  received  the  total 
scholarship  for  all  three  years. 


Table  4 


Participation  in  the  1990  Achievement  Tests  and  Diploma  Exams 


Grade 

Test 

District 

Province 

(%) 

(%) 

Grade  3 

Mathematics 

94.0 

88.3 

Grade  6 

Science 

94.4 

86.6 

Grade  9 

Language  Arts 

80.7 

90.7 

Grade  12* 

English  30 

64.2 

57.3 

English  33 

26.6 

21.5 

Social  Studies  30 

61.7 

48.0 

Mathematics  30 

54.5 

47.9 

Biology  30 

62.3 

47.1 

Chemistry  30 

47.4 

37.3 

Physics  30 

28.6 

19.9 

*    Grade  12  rates  are  based  on  the  total  students  writing  in  January  and  June  1990,  divided  by  the 

September  30, 1989  enrollment 
Source:  Alberta  Education 
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Implementation 


Administrators  considered  what  might  best  be  done  about  the  information  collected, 
related  literature  and  personal  perceptions,  and  what  plans  could  be  made  to  improve 
the  district.  Important  outcomes  for  administrators  were:  increased  awareness  of 
"what  exists"  in  Brooks;  a  vision  of  expected  student  behaviors  in  the  Brooks 
School  District;  ideas,  issues  and  problems  which  administrators  face  on  a 
day-to-day  basis,  that  is,  what  are  the  obstacles  or  impediments  to  attaining  the 
goal  regarding  student  behavior;  new  ideas  about  how  student  discipline  could  be 
handled,  on  a  school-wide  discipline  basis,  and  an  action  plan. 

The  action  plan  included  involving  the  community  in  establishing  reasonable 
expectations  of  children  of  different  ages  with  respect  to  conduct  and  values  at 
home  and  at  school.  A  series  of  meetings  was  organized  in  1992  in  order  to  collect 
information  and  establish  standards  to  involve  others  in  the  identification  of  what 
constitutes  good  behavior  and  what  steps  might  be  taken  to  promote  good  behavior. 

The  purpose  for  the  Character  in  our  Community  meetings  was  to  review  and 
reestablish  a  shared  vision  among  schools,  homes  and  community  with  respect  to 
developing  prosocial  attitudes  and  behaviors  among  our  youth.  Our  goal  was 
to  jointly  develop  a  set  of  beliefs  for  our  community. 

The  process  for  involving  the  community  included  setting  up  a  Steering  Committee 
consisting  of  the  superintendent,  three  school  administrators,  and  the  quality 
indicators  director.  This  committee  clarified  the  purpose  of  the  meetings  and  fine- 
tuned  the  process.  It  was  agreed  that  a  series  of  three  meetings  with  the  community 
would  help  assess  the  support  and  interest  that  exist  regarding  the  promotion  of 
prosocial  skills  in  our  youth.  It  was  also  agreed  that  there  would  be  a  field-test 
meeting  with  identified  School  Advisory  Council  participants  to  first  validate  the 
process  to  be  used. 

The  meetings  with  the  community  established  a  set  of  beliefs  as  well  as  a  set  of 
statements  of  personal  commitment  These  formed  the  basis  for  potential 
community  use,  and  reaffirmed  the  support  for  school  district  efforts  in  terms  of 
promoting  prosocial  skills.  These  belief  statements  are  presented  in  Figure  2. 


Conclusions 

In  our  efforts  to  determine  whether  we  have  the  "right"  blend  of  attention  to 
academic  success  and  attention  to  the  development  of  "character"  a  baseline  of 
information  was  established  for  the  Brooks  School  District.  It  included  how  the 
staff  and  administrators  viewed  the  district,  how  much  student  homework  was 
completed,  the  amount  of  vandalism  and  discipline  that  occurred,  how  the  former 
students  viewed  the  district  and  achievement  rates. 

Staff  and  administrators  in  the  Brooks  School  District  believe  that  students  behave 
responsibly.  This  is  shown  in  the  indicators  which  were  identified:  mutual  respect 
and  recognition,  pride  in  self,  others,  home,  school  and  community,  courteous, 
supportive  caring  and  sharing  attributes,  mutual  trust  and  open  communication  and 
high  expectations.  The  indicators  were  measured  through  homework  completion, 
vandalism,  discipline,  Former  Student  Survey  and  student  achievement  rates. 
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CHARACTER  IN  OUR  COMMUNITY  BELIEFS 

Character  is  the  ability  to  identify  the  elements  of  a  circumstance  or  situation,  to 
determine  one's  most  constructive  role  in  it,  to  carry  out  that  role  directly,  to  sustain 
it  as  long  as  necessary,  and  to  learn  from  the  consequences  of  one's  actions. 

Within  the  context  of  our  community  which  has  good  people  of  all  ages,  we  are 
committed  to  attaining  an  even  higher  standard  of  behavior  through  action 
consistent  with  the  following.  We  believe  that: 

1 .  Good  character  is  respecting  yourself  and  others.  No  one  has  the  right  to  take 
(without  consent),  harm  or  destroy  another  person's  physical  or  mental  well- 
being.  No  one  has  the  right  to  devalue  the  character  of  another. 

2 .  No  one  has  the  right  to  take  (without  consent),  harm  or  destroy  another 
person's  property. 

3 .  The  difference  between  rights,  responsibilities,  and  privileges  must  be  recognized. 

4.  The  home  must  provide  the  foundation  for  the  development  of  character. 

5 .  RulesAaws  need  to  be  fairly  and  consistently  applied. 

6.  Respect  is  a  mutual  endeavor  especially  between  adult  and  child. 

7 .  People  must  have  and  show  respect  for  rules  and  laws  in  the  community  and  for 
those  who  have  responsibility  for  enforcing  them. 

8 .  The  best  way  to  teach  character  is  to  be  a  good  role  model. 

9.  Positive  self-development  evolves  from  achieving  a  sense  of  purpose,  from 
having  a  positive  self-concept,  and  from  being  accepted. 

10.  Community  contributions  are  made  by  those  who  wish  to  institute  improvement 

11.  Young  people  need  and  want  age-appropriate  limits  and  expectations  on  their 
freedom  and  activities. 

12.  An  individual  is  accountable  for  personal  actions  and  must  live  with  the  consequences. 

13.  Appropriate  language  and  manners  are  expected  in  public  places. 

14.  Restricting  the  use  of  alcohol  and  drugs  to  responsible  use  is  necessary,  desirable  and 
achievable. 

15.  Social  activities  of  young  people  must  be  effectively  monitored  by  a  responsible  adult. 

16.  Offensive  behavior  will  not  be  tolerated  and  those  responding  appropriately 
deserve  the  support  of  the  community. 


Figure  2:  Community  Beliefs  in  Brooks 
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The  1990  homework  completion  rate  baseline  for  the  district  was  84.9%.  There  was 
no  vandalism  from  September  to  December  1990.  There  were  103  cases  of 
counseling  for  inappropriate  student  behavior  and  parental  contact  by  administrators 
during  a  two- week  collection  period. 

The  1991  Former  Student  Survey  indicated  that  attention  was  paid  to  academic 
success,  and  that  there  were  firm  behavioral  expectations.  However,  the  respondents 
felt  that  the  teachers  in  the  Brooks  School  District  need  to  pay  further  attention  to  the 
development  of  character  (especially  in  terms  of  promoting  respect,  tolerance, 
recognition  and  guidance).  Student  achievement  was  relatively  high  on  provincial 
exams  in  1989-90. 


Implications 

The  Brooks  School  District  personnel  were  concerned  that  students  behave  in  order 
for  learning  to  occur.  It  is  timely  in  that  throughout  Alberta  and  western  Canada 
there  is  an  increasing  number  of  students  with  behavioral  difficulties.  The  efforts 
made  by  our  school  district  in  the  promotion  and  support  for  positive  behavioral 
outcomes  could  be  adapted  in  other  school  districts.  It  is  important  to  remember 
that  student  behavior  is  first  and  foremost  determined  by  the  family,  and  it  is 
therefore  important  to  establish  a  school-family-community  partnership. 

One  of  the  expected  long-term  outcomes  of  the  Brooks  project  was  higher 
satisfaction  among  students,  graduates  and  employees.  This  will  need  to  be 
monitored  over  time  and  compared  to  the  original  results.  While  a  higher  than 
normal  percentage  of  discipline  related  cases  was  expected,  this  did  not  prove  to  be 
the  case,  but  needs  to  be  monitored  over  time.  Dropout  rates  were  not  addressed 
during  the  project,  as  there  was  no  agreement  reached  regarding  a  definition. 

Maintenance  of  high  academic  achievement  test  and  diploma  exam  results, 
participation  rates  and  Rutherford  Scholarships,  and  wider  acceptance  of  the  quality 
indicators  developed  by  stakeholders  in  Brooks  were  expected  long-term  results. 
These  also  need  to  be  monitored  over  time  now  that  measures  have  been 
established. 


Recommendations 

There  is  increasing  awareness  by  students,  parents  and  teachers  of  the  importance 
of  good  behavior  for  students  to  learn.  This  project  looked  at  what  good  student 
behavior  is.  We  have  strong  support  from  the  staff  of  the  Brooks  School  District 
The  objective  of  the  stakeholders  is  to  positively  influence  student  behavior  through 
collaborative  community  efforts. 

It  is  recommended  that  other  districts  concerned  with  promoting  positive  student 
behavior  adopt  and  modify  measures  to  suit  their  school  district.  These  might 
include:  staff  questionnaires,  administrator  interviews,  student  homework 
completion  rates,  vandalism  rates,  discipline  rates,  former  student  surveys  and 
telephone  interviews.  It  is  also  recommended  that  school  districts  form 
partnerships  between  their  schools,  parents  and  communities.  It  is  important  for 
schools  to  have  some  representation  (in  an  unofficial  capacity)  with  the  parents  in 
order  to  work  together  in  an  effort  to  promote  responsible  student  behavior. 
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For  Alberta  Education  it  is  recommended  that: 

•  the  information  from  the  Brooks  Project  be  shared  with  interested  school  districts 
throughout  the  province; 

•  support  be  made  available  to  provide  for  further  collaboration  and  follow-up 
among  projects  with  common  interests;  and 

•  further  research  projects  be  initiated  by  the  province  so  that  Alberta  can  maintain 
its  place  at  the  forefront  of  efforts  to  improve  the  quality  of  education. 


Follow-up 

The  Character  in  our  Community  (CIC)  formed  a  follow-up  committee  under  the 
auspices  of  two  school  administrators.  The  purpose  of  this  committee  was  to 
promote  acceptable  young  people's  behavior  throughout  the  community.  The 
School  Advisory  Councils  (SAC)  in  each  school  have  a  representative  on  the 
committee.  It  will  be  their  responsibility  to  communicate  committee  initiatives  to 
their  school  SAC. 

The  school  district  has  a  portfolio  of  information  collected  at  a  point  in  time,  such  as 
discipline  rates,  which  could  easily  be  duplicated.  Serious  consideration  has  also 
been  given  to  surveying  students  after  they  graduate  on  a  yearly  basis  to  monitor  the 
success  rate  regarding  the  promotion  of  positive  student  behavior  over  time. 


Concluding  Statement 

Our  original  premise  was  that  students  need  to  behave  in  order  to  learn.  We  found 
that  students,  families  and  the  community  have  the  same  belief.  The  information 
collected  indicates  how  successful  we  were  in  terms  of  our  combined  efforts.  The 
Brooks  EQI  Project  sought  to  establish  whether  or  not  the  "right"  blend  of  attention 
to  academic  success  and  "character"  within  the  framework  of  a  firm  but  supportive 
set  of  behavioral  expectations  and  policies  was  implemented.  To  this  end,  we 
reviewed  the  present  policies  as  well  as  collected  information  from  parents, 
students,  and  teachers  on  responsible  student  behavior  and  academic  success. 
The  collection  process  has  heightened  the  awareness  of  staff  and  community 
regarding  student  behavior.  The  belief  has  been,  and  still  is,  that  we  have  a  good 
school  district  that  promotes  positive  student  behavior.  The  Brooks  School  District 
has  identified  a  set  of  indicators  to  measure  this  belief. 

There  is  a  strong  interest  in  promoting  a  positive  community  attitude  in  developing 
responsible  student  behavior.  The  district  will  direct  its  efforts  to  monitoring 
positive  student  behavior,  and  will  continue  to  heighten  the  awareness  of  the  staff 
and  community  as  it  applies  to  positive  student  outcomes.  It  is  important  that  any 
school  district  be  committed  to  involving  the  community  in  efforts  to  promote 
responsible  student  behavior  since  much  time  and  energy  is  involved.  Increasing 
efforts  will  be  required  by  educators  to  support  the  family  and  community  to 
promote  responsible  student  behavior. 
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A  Study  to  Identify  and  Measure 
Desirable  Student  Social  Outcomes 


Fort  McMurray  School  District  No.  2833 


This  report  outlines  one  school  jurisdiction's  efforts  to  develop  a  set  of  social 
skills,  behaviors,  attitude  indicators  and  standards  that  can  be  used  to  assess  the 
quality  of  a  selected  portion  of  that  jurisdiction's  educational  program  and 
delivery.  It  describes  the  development  of  an  expanded  systems  model  of  social 
competence,  the  use  of  standardized  instruments  and  finally  the  development  of  a 
three-level  approach  to  measuring  and  reporting  social  skills  development. 


Introduction 

Rationale 

The  focus  on  assessing  and  reporting  student  social  skills  reflected  the  district's 
recognition  of  the  broadening  mandate  of  schools  in  modern  society.  Schools  are 
becoming  increasingly  responsible  for  the  broad  education  of  students  that  now 
includes  a  social  dimension.  Despite  an  extensive  body  of  research  on  social 
competence,  and  its  suggested  significant  role  in  student  academic,  social  and 
psychological  development,  current  definitions  of  social  competence,  as  well  as 
strategies  for  its  measurement  and  reporting  in  the  school  environment,  are 
limited.  In  recognition  of  this,  and  as  part  of  the  Educational  Quality  Indicators 
initiative,  the  Fort  McMurray  School  District  No.2833  undertook  a  study  to 
examine  the  measurement  of  student  social  competence.  The  major  purpose  of 
the  three-year  project  was  to  develop  a  set  of  social  skills,  behaviors  and  attitude 
indicators  and  standards  that  could  then  be  used  to  assess  the  quality  of  a  selected 
portion  of  its  educational  program  and  delivery. 


Purposes  of  the  Study 

In  order  for  the  study  to  meet  the  purposes  outlined  above,  the  following  tasks 
were  completed: 

1.  identify  or  develop  a  set  of  desirable  student  social  skills; 

2.  identify  or  develop  measurable  outcomes  and  standards  for  the  desirable 
student  social  skills; 


Social  Outcomes 


122 


Fort  McMurray  Public 


3.  establish  a  methodology  for  collecting,  analyzing  and  interpreting  the  data; 

4.  identify  or  develop  strategies  for  teaching  the  identified  desirable  student 
social  skills; 

5.  identify  or  develop  a  means  of  reporting  information  and  findings  to  users. 


A  Model  of  Social  Competence 

Social  skills  are  a  part  of  human  behavior  that  are  hard  to  define  and  measure.  A 
model  helps  to  conceptualize  and  understand  the  relationships  among  parts.  The 
Fort  McMurray  Public's  EQI  Project  used  a  systems  perspective  based  on  a 
modified  CIPP  model  (Worthen  &  Sanders,  1987)  as  the  conceptual  framework. 
The  systems  perspective  was  used  as  the  basic  organizational  framework  for  the 
project  because  it  "was  compatible  with  the  beliefs  of  the  study  team  and  appeared 
to  provide  good  information  for  program  development  and  student  evaluation 
related  to  social  competence"  (Young,  1990,  p.  2). 

Certain  commonalities  are  shared  among  the  theories  and  models  of  social  skills 
and  social  competence. 

1.  There  are  specific  elements  and  skills. 

2.  There  is  a  need  for  the  elements/skills  to  be  processed  in  an  appropriate 
manner;  the  processes  may  be  either  cognitive  or  affective. 

3.  The  appropriateness  of  behaviors  to  specific  situations  is  crucial. 

4.  There  is  no  agreed  upon  generic  social  skills  listing.  Also,  any  list  needs  to 
be  flexible  to  some  extent  to  the  needs  of  the  user. 

5.  Appropriate  social  responses  are  affected  by  the  developmental  level  of  the 
respondent,  and  the  audience  for  the  respondent. 

6.  Social  responsibility  can  be  learned. 

7.  The  whole  social  competencies  construct  is  dynamic,  synergistic,  and 
organic. 

8.  Specific  social  elements  and  skills  are  comprised  of  specific  and  discrete 
verbal  and  nonverbal  behaviors  and  entail  both  effective  and  appropriate 
initiations  and  responses. 

9.  The  skills  are  interactive  by  nature. 

10.  Deficits  and  excesses  in  social  response  can  be  specified  and  targeted  for 
intervention. 

11.  Instructors  attempting  to  teach  social  skills  or  appropriate  social  responses 
should  adhere  to  the  "relevance  of  behavior  rule"  (Allyon  &  Azrin,  1968) 
which  states  that  instructors  should  teach  only  those  behaviors  that  will 
continue  to  be  naturally  reinforced  after  training.  Adopting  this  viewpoint 
assists  in  ensuring  that  social  skills  selected  for  instruction  will  have  some 
intrinsic  value  to  the  child,  some  benefits  for  the  child,  and  be  valued  by 
others  who  would  likely  reward  their  occurrence  (Young,  1990,  p.  4). 

The  systems  model  of  social  competence  was  a  compilation  of  a  number  of 
research-based  ideas  about  the  process  (Perry  &  Perry,  1987;  Hargie  &  McCartan, 
1986)  and  the  factors  (Reschly  &  Gresham,  1981)  which  influence  an  individual's 
social  interaction  skills.  The  social  outcomes  deemed  desirable  were  those 
outlined  in  the  1988  Alberta  School  Act  (Section  7,  p.  11)  and  in  the  1989  Guide 
to  Education.  Figure  1  presents  the  model. 
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Measuring  Social  Skills 


The  conceptualization  of  social  competence  within  a  systems  framework  resulted 
in  the  identification  of  three  major  facets  of  the  concept:  social  skills,  social 
information  processing  and  socially  desirable  outcomes.  These  initial  three  facets 
of  the  social  competence  model  were  within  a  situational  context. 

Given  the  preliminary  definition  of  social  competence,  and  the  identification  of  a 
broader  system  framework  in  which  to  approach  its  teaching  and  measurement, 
the  study  team  addressed  the  specific  task  of  developing  an  assessment 
methodology  for  measuring  and  reporting  student  social  competence.  Drawing 
on  the  findings  of  the  literature  review,  which  indicated  the  complexity  and  time- 
consuming  nature  of  developing  reliable  and  valid  social  skills  measurement  and 
reporting  systems,  the  study  team  decided  to  examine  existing  instruments  to 
determine  their  appropriateness  for  use  in  the  Fort  McMurray  context.  Potential 
instruments  were  reviewed  that  fit  the  following  criteria: 

1.  The  instrument  focused  on  students  in  regular  classrooms. 

2.  The  instrument  accommodated  a  variety  of  developmental  levels,  largely 
defined  as  divisions  I  (grades  1  to  3),  II  (grades  4  to  6),  in  (grades  7  to  9),  and 
IV(grades  10  to  12). 

3.  The  instrument  included  a  variety  of  sub-scales  and  items  that  addressed  the 
social  response  areas  identified,  and  had  the  potential  to  be  adapted  to  meet 
the  requirements  of  the  team's  particular  needs. 

4.  The  instrument  focused  on  positive  social  behavior,  as  opposed  to  negative 
social  behavior,  or  lack  of  social  behavior. 

5.  The  instrument  included  variations  pertinent  to  the  various  audiences  who 
might  be  asked  to  respond  such  as  parents,  teachers  and  students. 

The  study  team  identified  two  instruments  having  the  potential  to  assess  student 
social  skills,  particularly  perceptions  -  the  Self -Perception  Profile  for  Children 
(1985)  and  the  Self -Perception  Profile  for  Adolescents  (1988);  both  developed  by 
Susan  Harter.  Subsequent  consultations  with  the  author  confirmed  the  potential  of 
both  instruments  for  use  as  part  of  the  research  project  given  their  capacity  for 
adaptation.  A  third  instrument,  the  Teacher  Rating  of  Social  Skills  -  Children 
(TROSS-C)  developed  by  Clarke,  Gresham  and  Elliott  (1985),  was  also  identified 
as  an  appropriate  measure  for  observation  of  student  behavior.  The  TROSS-C  was 
revised  by  Gresham  and  Elliott  (1990)  based  on  further  development  work,  and 
renamed  the  Social  Skills  Rating  System  (SSRS).  The  revisions  to  the  instrument 
did  not  substantively  alter  its  form  or  content. 

As  a  further  assessment  of  their  suitability,  the  selected  instruments  were  cross- 
referenced  to  the  desired  social  outcomes  developed  as  part  of  the  model  of  social 
competence  (Table  1).  The  results  of  this  cross-referencing  exercise  show  the 
instruments  to  provide  good  coverage  of  the  theoretical  construct. 


Social  Outcomes 


125 


Fort  McMurray  Public 


Table  1 

Cross-Referencing  Social  Outcomes  and  Assessment  Instruments 


Desirable  Social  Outcomes  1 


Instruments 


1.  BEHAVIORS: 

1.1  basic  interaction  skills  1.1 

1.2  coping  skills  1.2 


1.3    responding  skills  1.3 


1.4    classroom/academic  skills  1.4 


SSRS  2  (Social  Initiation) 
TROSS-C 3  (Academic 
Performance,  Peer 
Reinforcement,  Cooperation); 
TROSS-C  (Peer 
Reinforcement,  Academic 
Performance) 
TROSS-C  (Academic 
Performance) 


PERCEPTIONS: 

2.1  goals/motivation/needs 


2.2  appearance 

2.3  self-esteem 


2.1 


2.2 
2.3 


SPP  4  (for  Adolescents  -  Job 
Competence,  Romantic  Appeal, 
Friendship) 

SPP 5  (Physical  Appearance) 
SPP  (Self-Esteem) 


SOCIAL  APPROPRIATENESS: 


3.1  acceptance 

3.2  behavior  conduct 

3.3  positive  reinforcement 


3. 1  SPP  (Social  Acceptance) 

3.2  SPP  (Behavior  Conduct) 

3.3  SPP  (Peer  Reinforcement) 


1  Appropriate  desirable  social  outcomes  at  various  developmental  levels  and  as  outlined  in  the 
Alberta  School  Act  (1988)  and  the  Guide  to  Education  (1989). 

2  Social  Skills  Rating  System  (Gresham  &  Elliott,  1990). 

3  Teacher  Rating  of  Social  Skills  -  Children  (Clarke,  Gresham  &  Elliott,  1985). 

4  Self-Perception  Profile  for  Adolescents  (Harter,  1988). 

5  Self-Perception  Profile  for  Children  (Harter,  1985)  and  Self-Perception  Profile  for 
Adolescents  (Harter,  1988). 
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Field  Test  of  Standardized  Instruments 

In  the  second  phase  of  the  project,  the  three  identified  instruments  were  field- 
tested  in  the  fall  of  1990.  The  major  objectives  of  the  field  test  were  to  confirm  the 
quality  of  the  data  gathered  by  the  instruments,  and  to  review  the  ease  and 
practicality  of  instrument  administration.  The  appropriate  SSRS  instrument  was 
administered  to  a  grade  3  class  and  a  grade  1 1  (semestered)  class;  the  SPP-C  to  a 
grade  6  class  and  the  SPP-A  to  a  grade  1 1  (semestered)  class.  Prior  to  the  field 
test,  teachers  were  given  a  brief  overview  of  the  purpose  of  the  study,  as  well  as  a 
summary  of  the  instruments  to  be  used.  As  part  of  the  instrument  administration, 
teaching  staff  were  asked  to  complete  one  form  for  each  student,  have  the  student 
complete  one,  and  have  the  parent  complete  one.  One  follow-up  was  completed 
for  parents  who  did  not  return  the  forms.  Once  the  completed  forms  were  returned 
from  the  three  groups  of  participants,  an  analysis  was  completed  for  each  student 
in  the  sample. 

Completed  assessments  (teacher,  student  and  parent  instruments)  were  collected 
for  34  of  the  84  students  assessed  (a  completed  sample  of  40%).  Two  major 
problems  were  identified  through  the  administration:  training  in  test 
administration  for  teachers;  and  ease  of  instrument  administration.  Teachers 
reported  that  they  felt  uncomfortable  both  in  administering  the  instrument,  and 
in  analyzing  and  interpreting  the  results.  They  further  identified  some  potential 
suspicion  and  antagonism  among  parents  relating  to  the  process.  Teachers 
suggested  the  need  for  the  introduction  of  comprehensive  training  sessions  if 
the  instruments  were  intended  to  become  a  standard  component  of  student 
assessments.  Teachers  did  acknowledge,  however,  the  value  of  the  information 
gathered  by  the  individual  instruments. 

Teachers  deemed  administration  difficult  because  of  the  amount  of  time  involved 
for  each  teacher  both  in  assessing  individual  students,  as  well  as  ensuring  parent 
returns.  For  the  field  tests,  parental  returns  were  low  (34  out  of  a  possible  84) 
with  five  outright  refusals.  Teachers  also  reported  that  the  administration  of  the 
instruments  in  a  semestered  high  school  in  November,  was  too  early  in  the  school 
term  for  them  to  make  informed  assessments  of  students'  social  skills. 

Based  on  the  results  of  the  field  test  of  these  instruments,  all  three  were 
considered  "impractical"  for  use  with  whole  classes  as  part  of  a  universal  process 
of  student  social  competence  assessment  and  reporting  -  although  their  use  with 
smaller  groups  of  students  was  considered  feasible.  In  their  discussions  of  the 
field-test,  teachers  suggested  the  development  of  a  simpler-to-administer  Social 
Competence  Checklist,  with  the  use  of  commercial  tests  by  counselors  as  a  "back- 
up" to  the  simpler  checklist,  and  it  was  to  this  strategy  that  the  project  team  turned 
next. 


A  Three-Level  Approach  to  Measuring  Social  Skills 

From  the  field  test  of  the  instruments  and  subsequent  discussions,  a  three-level 
approach  to  measurement  evolved,  focusing  primarily  on  issues  of  practicality, 
administration  and  quality  of  information.  The  latter  referred  to  the  original 
instrument  selection  criteria.  The  steering  committee  began  to  develop  a  checklist 
to  help  teachers  and  parents  more  easily  identify  children's  level  of  social  skills. 
The  checklist  was  based  on  the  same  criteria  to  which  the  standardized 
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instruments  had  been  cross-referenced  (Table  1).  It  attempted  to  measure  those 
skills  which  would  result  in  desirable  student  social  outcomes.  The  checklist 
eventually  became  the  Social  Skills  Diagnostic  Screen. 

Additionally,  district  report  cards  were  undergoing  a  detailed  review.  The  portion 
of  the  report  cards  where  teachers  comment  on  children's  social  skill  development 
was  revised  to  reflect  a  more  detailed  and  research-based  assessment  using 
selected  components  of  the  Social  Skills  Diagnostic  Screen. 

Level  One  -  Each  teacher  reported  to  parents,  at  report  card  time,  the  social 
development  of  their  children.  Each  teacher  rated  students  as  being  well 
developed,  developing  appropriately,  or  needing  further  development.  The 
ratings  were  based  on: 

•  personal  factors 

•  interpersonal  factors 

•  social  reasoning  skills 

•  classroom  behaviors 

Level  Two  -  If  any  of  the  students,  parents,  or  teachers  wished  or  required  a  more 
detailed  assessment,  the  District's  Social  Skills  Diagnostic  Screen  would  be 
completed  by  the  teacher  for  use  in  providing  a  rating  for  the  report  card,  forming 
the  basis  for  discussion  with  parents  and/or  the  student,  or  for  developing  a 
moderate  program  of  remedial  action.  Additionally,  a  form  could  be  completed 
by  the  student,  the  parents,  and  a  Composite  Student  Profile  compiled  to  be  used 
as  the  basis  for  discussion.  Using  this  information,  a  program  of  intervention  or 
skill  development  exercises  could  be  initiated. 

The  Social  Skills  Diagnostic  Screen  consists  of  four  sections:  personal  factors, 
interpersonal  factors,  social  reasoning  skills,  and  classroom  behaviors  which 
represent  more  expansive  investigations  of  the  same  four  areas  of  reporting 
identified  at  level  one.  Each  section  contains  a  series  of  simple  to  administer  and 
score  questions.  The  screen  may  be  administered  in  either  individual  or  group 
settings,  and  may  be  completed  by  the  student,  a  student's  peer,  the  teacher,  a 
parent  or  any  other  individual  known  to  the  subject  in  other  than  a  casual  manner. 
The  screen  is  designed  to  be  applicable  to  school-aged  subjects,  across  all  grade 
levels.  The  time  taken  to  administer  the  screen  varies  from  approximately  10  to 
15  minutes  with  older  subjects,  to  20  to  30  minutes  with  younger  subjects  who 
may  need  to  be  interviewed  on  an  individual  basis. 

Level  Three  -  If  detailed  knowledge  of  the  student's  social  development  was 
required,  the  student's  case  would  be  referred  to  the  counselor  for  further 
assessment  and  the  development  of  an  Individualized  Program  Plan  related  to 
the  development  of  appropriate  and  acceptable  social  behavior.  The  battery  of 
assessment  instruments  to  be  used  may  include: 

•  Social  Skills  Rating  System  (Gresham  &  Elliott,  1990) 

•  Self -Perception  Profile  for  Adolescents  (Harter,  1988) 

•  Self-Perception  Profile  for  Children  (Harter,  1985) 

Counselors  would  be  trained  in  the  assessment  instruments  and  the  related 
procedures  through  inservices  provided  by  district  consultants. 
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Findings 


The  major  outcomes  of  the  research  were  twofold:  first,  the  conceptualization  of 
the  systems  model  of  social  competence  that  drew  on  existing  literature  relating 
to  the  concept  and  provided  a  framework  for  its  definition  and  subsequent 
measurement;  and  second,  the  development  of  a  three  level  approach  to  assessing 
and  reporting  student  social  competence  development  within  the  school 
environment.  As  part  of  this  three-level  approach,  the  Social  Skills  Diagnostic 
Screen,  was  developed  and  subsequently  pilot-tested.  In  addition  to  the 
development  of  this  instrument,  an  activity  manual  for  teaching  social  skills 
was  prepared,  for  use  in  conjunction  with  the  screen. 

The  development  of  the  social  competence  model,  and  the  associated  assessment 
methodology  was  completed  as  part  of  a  series  of  stages.  Initially,  the  concept  of 
social  competence  was  defined  based  on  the  findings  of  a  substantive  literature 
review  and  subsequent  discussions  of  the  concept  by  the  project  study  team.  As  part 
of  the  second  phase,  standardized  social  skill  measurement  instruments  were  pilot- 
tested  to  determine  their  appropriateness  for  use  in  the  school  environment.  As  a 
result  of  the  findings  of  these  pilot  tests,  a  more  flexible  and  comprehensive 
assessment  model  was  introduced  that  included  a  three-level  assessment  protocol 
that  moved  from  the  general  to  the  specific.  The  second  level  of  the  model  included 
a  specially  developed  assessment  instrument,  the  Social  Skills  Diagnostic  Screen. 

The  findings  of  the  three-year  research  project  suggest  the  value  of  the  three-level 
approach  to  measuring  and  reporting  student  social  competence.  Field-testing  of 
standardized  commercial  instruments  revealed  considerable  difficulty  in  their 
practical  application  in  the  everyday  school  setting  -  given  both  the  unfamiliarity 
of  teachers  with  the  instruments  themselves,  as  well  as  their  unwieldy  and 
complex  nature  that  was  reflected  in  the  time  taken  to  administer  them.  By 
contrast,  the  Social  Skills  Diagnostic  Screen,  developed  in  part  from  a  review  of 
existing  instruments,  proved  more  practical  for  classroom  use,  and  in  pilot  testing 
demonstrated  some  measure  of  statistical  validity  (content)  and  reliability  (test- 
retest).  The  results  of  the  limited  pilot  testing  of  both  the  screen  and  the  broader 
three-level  approach  to  assessing  social  skills,  suggested  their  value  as  part  of  an 
integrated  assessment  and  reporting  package. 


Conclusions 

The  findings  of  the  project  cast  considerable  light  on  the  complex  nature  of 
measuring  and  reporting  student  social  competence.  Challenges  similar  to  those 
identified  within  the  literature  were  initially  encountered  in  defining  social 
competence.  The  adoption  of  a  systems  perspective  provided  a  broad  framework 
for  the  conceptualization  of  social  competence.  It  provided  the  framework  for  the 
inclusion  of  the  context,  input,  process  and  product  dimensions  essential  to  any 
model  of  social  competence.  As  a  theoretical  concept  and  practical  component 
of  everyday  life  and  interactions,  social  competence  retains  an  inherent  dynamic 
quality  that  mitigates  against  its  simple  description.  Rather,  the  concept  needs  to 
be  expressed  in  'fluid*  terms  that  reflect  its  dynamic,  and  complex  nature,  as  well 
as  its  link  to  day-to-day  interactions. 
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Given  the  challenge  in  defining  social  competence,  as  well  as  the  specific 
elements  that  contribute  to  it,  the  current  project  also  revealed  the  complexity  of 
its  subsequent  measurement  and  reporting.  The  initial  field  test  of  standardized 
commercial  instruments  failed  to  meet  the  project  need  for  a  practical  assessment 
methodology  that  provided  the  basis  for  meaningful  dialogue  between  teachers 
and  parents  on  student  social  development.  The  subsequent  development  of  a 
three-level  approach  represented  a  creative  and  innovative  strategy  that,  based  on 
limited  field  testing,  proved  practical,  valid  and  reliable  in  the  measurement  and 
reporting  of  student  social  competence.  While  this  approach  performed 
satisfactorily  within  the  pilot  test,  it  requires  further  review  and  testing  to  confirm 
its  validity,  reliability  and  practicality  as  an  assessment  and  reporting  strategy. 


Implications 

The  conceptual  model  of  social  competence  developed  within  the  current  study,  as 
well  as  the  three-level  approach  to  assessment  designed  to  accompany  it,  both 
have  potential  for  application  within  the  school  environment.  Specifically,  the 
three-level  approach  is  research-based  and  cross-referenced  to  desirable  student 
outcomes,  and  thus,  may  be  effectively  used  in  varied  school  settings  for  assessing 
and  reporting  student  social  skills. 

Empirically,  the  development  of  a  valid,  reliable  and  practical  assessment  and 
reporting  methodology  for  student  social  competence  is  an  important  break- 
through. While  several  standardized  commercial  instruments  are  available,  their 
application  in  the  school  environment  is  limited  given  their  complexity  in  terms  of 
administration  and  practicality  for  the  classroom  teacher. 

As  an  extension  of  the  advantage  gained  through  the  development  of  a  practical 
assessment  and  reporting  methodology,  it  may  be  anticipated  that  teachers  will  be 
able  to  address  instruction  in  social  skills  more  effectively  and  confidently.  Not 
only  will  teachers  be  in  a  position  to  determine  the  student  level  of  social 
competence,  but  they  will  also  be  able  to  review  possible  causes  of  skill  deficits, 
as  well  as  the  impact  of  specific  interventions  designed  to  address  deficits.  In  the 
long  term,  the  development  of  an  effective  assessment  and  reporting  strategy  may 
thus  impact  on  the  nature  and  content  of  social  skills  instruction  in  the  classroom 
and  school  environments.  Within  this  context,  the  Social  Skills  Development 
Activity  Manual  (Sherman,  1991)  developed  in  conjunction  with  the  Social  Skills 
Diagnostic  Screen,  serves  as  a  valuable  resource  for  teachers  and  parents. 


Recommendations 

Based  on  the  preliminary  findings  of  the  research  to  date,  the  major 
recommendation  of  the  current  study  is  that  research  continue  in  measuring  and 
reporting  social  competence.  Student  social  competence  forms  a  key  element  of 
general  educational  development,  and  one  that  has  frequently  been  overlooked 
given  problems  associated  with  both  its  definition  and  measurement  in  the  school 
environment.  Despite  the  encouraging  results  of  the  current  research,  more 
extensive  field  testing  of  the  Social  Skills  Diagnostic  Screen  is  required.  Scoring 
norms  need  to  be  established  for  different  user  groups,  as  well  as  for  individual 
sections  of  the  screen  so  that  it  may  be  more  effectively  used  to  pinpoint 
connections  and  specific  areas  requiring  further  assessment.  More  research  is 
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also  required  into  the  use  of  interventions  to  address  student  skill  deficits. 
Specifically,  the  nature  and  extent  of  interventions  needs  to  be  clarified  for 
individual  students  based  on  their  individual  needs. 

To  assist  in  this  broader  testing  of  the  three-level  approach,  as  well  as  the  screen 
itself,  it  is  recommended  that  both  be  field-tested  in  a  variety  of  school  settings,  as 
well  as  individual  settings  in  long-term  longitudinal  studies.  The  findings  from 
these  more  extensive  field  tests  should  yield  valuable  results  regarding  the 
validity,  reliability  and  practicality  of  the  assessment  process. 


Follow-up 

Within  the  Fort  McMurray  context,  follow-up  to  the  project  will  be  ongoing. 
Parents  have  requested  additional  information  on  strategies  they  may  use  with 
their  own  children  to  work  on  appropriate  social  skill  development.  Similarly, 
professional  and  support  staff  from  across  the  school  district  have  expressed  the 
wish  both  to  facilitate  and  take  part  in  initiatives  relating  to  the  whole  area  of 
social  skills  development  and  assessment.  At  the  district  level,  the  board  has 
determined  that  student  social  competence  is  a  priority  for  the  1992-1993  school 
year.  As  part  of  this,  the  three-level  approach  will  become  an  integral  component 
of  the  policy  and  practice  of  the  district. 


Concluding  Statement 

The  model  of  social  competence  developed  within  the  current  study,  and  its 
extension  into  a  three-level  approach  to  assessing  and  reporting  for  use  in  the 
school  environment,  forms  an  important  initiative  in  the  ongoing  review  of 
educational  quality.  The  assessment  and  reporting  of  student  social  competence 
within  the  school  context  has,  to  date,  been  limited,  with  educators  forced  to  rely 
on  standardized  commercial  instruments  that  frequently  do  not  meet  either  the 
needs  of  students  or  staff.  The  need  for  students  to  develop  an  appropriate  level 
of  social  competence  is  critical,  given  its  close  relationship  with  other  areas  of 
educational  performance  and  growth. 

The  three-level  approach  to  measuring  social  skills,  and  its  second-level  Social 
Skills  Diagnostic  Screen,  form  one  response  to  the  need  for  positive  indicators  and 
standards  that  address  student  social  skills  in  the  school  environment.  This  project 
has  focused  on  the  development  of  an  assessment  model  that  is  straightforward  to 
use  and  practical.  If  the  research  has  raised  an  awareness  of  the  importance  of 
social  skill  acquisition  as  it  relates  to  student  development,  then  a  further  goal  has 
been  met. 

References 

Alberta  Education.  (1989).  Guide  to  Education:  Junior  High  School  Handbook. 
Edmonton:  Alberta  Education. 

Allyon,  T.,  &  Azrin,  N.  (1968).  The  Token  Economy.  New  York:  Appleton- 
Century-Crofts. 


Social  Outcomes 


131 


Fort  McMurray  Public 


Clarke,  L.,  Gresham,  F.M.,  &  Elliott,  S.N.  (1985).  Development  and  validation  of 
a  social  skills  assessment  measure:  The  TROSS-C.  Journal  of 
Psychoeducational  Assessment,  4,  347-356. 

Gresham,  F.M.,  &  Elliott,  S.N.  (1990).  Social  Skills  Rating  System.  Circle  Pine, 
MN:  American  Guidance  Service. 

Hargie,  O.,  &  P.  McCartan  (1986).  Social  skills  training  and  psychiatric  nursing. 
London:  Croom  Helm. 

Harter,  S.  (1985).  Manual  for  the  Self -Perception  Profile  for  Children.  Denver: 
University  of  Denver. 

Harter,  S.  (1988).  Manual  for  the  Self -Perception  Profile  for  Adolescents. 
Denver:  University  of  Denver. 

Perry,  D.G.,  &  Perry,  L.C.  (1987,  July).  Applications  of  Dodge's  social 

information  processing  model  of  social  competence  to  the  study  ofprosocial 
behavior  in  children.  Paper  presented  at  the  biennial  meeting  of  the 
International  Society  for  the  Study  of  Behavioral  Development,  Tokyo,  Japan. 

Province  of  Alberta.  (1988).  School  Act.  Edmonton:  Queen's  Printer. 

Reschly,  D.L.,  &  Gresham,  F.M.  (1981).  Use  of  social  competence  measures  to 
facilitate  parent  and  teacher  involvement  and  non-biased  assessment.  Iowa: 
Iowa  State  University. 

Sherman,  A.  (1991).  Social  skills  development  activity  manual.  Fort  McMurray 
Public  School  District  No.  2833. 

Worthen,  B.R.,  &  Sanders,  J.R.  (1987).  Educational  evaluation:  Alternative 
approaches  and  practical  guidelines.  New  York:  Longman. 

Young,  D.  (1990,  June).  Measuring  social  competence  in  students.  Paper 
presented  at  the  annual  meeting  of  the  Canadian  Educational  Researchers' 
Association,  Victoria,  BC. 


Social  Outcomes 


132 


Fort  McMurray  Public 


Signs  of  Learning  in  the  Affective  Domain 


Lethbridge  RCSSD  No.  9 


The  Lethbridge  Catholic  Schools  Project  recognizes  the  importance  of  learning 
outcomes  in  the  affective  domain  and  the  efforts  of  teachers  to  comply  with 
assessing  and  reporting  those  outcomes.  The  project  was  designed  to  operate 
within  the  classroom  setting  and  to  be  subject  to  its  limitations.  This  report  presents 
the  background,  the  methodology  of  the  project,  the  findings  and  their  implications. 


Introduction 

The  Lethbridge  Catholic  Schools  researched  signs  of  learning  in  the  affective 
domain  as  part  of  the  Educational  Quality  Indicators  collaborative  initiative  to 
develop  an  indicator  system  which  will  provide  information  about  education 
to  Alberta  Education. 


Purposes 

The  first  purpose  of  this  study  was  to  find  indicators  to  identify  attitudes  in  the 
affective  domain  that  are  set  out  by  Alberta  Education  in  the  goals  of  schooling. 

A  second  purpose  was  to  devise  a  system  for  monitoring  and  recording  behaviors 
which  are  the  expression  of  the  attitudes. 

The  third  purpose  of  the  study  was  to  assist  educators  and  parents  in  their  efforts  to 
teach  and  model  behaviors  which  express  positive  attitudes  and  which  assist  in  the 
formation  of  positive  attitudes.  The  formation  of  positive  attitudes  is  an  important 
goal  of  education,  and  educators  are  expected  to  teach  and  to  encourage  growth  in 
positive  attitudes. 


Background 

Lethbridge  is  a  city  of  60,000  in  southwestern  Alberta.  Education  has  been  shared 
for  the  past  100  years  between  the  Public  District  and  the  Catholic  District  The 
Lethbridge  Catholic  District  has  one  high  school,  one  junior  high  school,  and  six 
elementary  schools  with  170  teachers  and  70  support  staff  to  serve  2,700  students. 
The  schools  are  places  of  hope  where  students  are  affirmed.  The  schools  aim  to 
provide  excellent  academic  education  in  an  environment  of  love  and  respect. 
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The  importance  of  the  affective  domain  to  Lethbridge  Catholic  Schools  is 
documented  in  its  policies  which  aim:  to  create  a  positive  climate;  to  help  shape 
wholesome  self-images  in  students;  to  build  community  through  modeling;  to  set 
guidelines  for  spiritual  growth  and  development;  to  acknowledge  the  dignity  of 
helping  children  grow  according  to  their  abilities;  to  instill  a  responsible  attitude 
toward  the  world  and  its  people;  to  teach  ways  to  relate  acceptably  to  others;  and  to 
guide  the  process  of  evaluation,  provide  a  sense  of  hope  and  success,  recognize  the 
need  for  continuous  evaluation  and  trace  the  steps  to  take  in  order  to  celebrate 
learning.  It  is  from  this  background  that  the  Lethbridge  Catholic  School  District 
came  to  the  decision  to  search  for  signs  of  learning  in  the  affective  domain. 


Scope 

This  project  relates  specifically  to  signs  of  learning  in  the  affective  domain.  It 
defines  the  affective  domain  as  that  area  of  student  development  which  includes 
attitudes,  interests,  appreciation,  feelings,  emotions  and  predispositions.  This 
project  makes  no  claim  to  finding  signs  of  learning  that  relate  to  cognitive  or  skill 
development,  however  crossover  may  occur. 


Assumptions 

The  project  was  based  on  the  following  assumptions  and  beliefs  which  both  limited 
and  directed  the  research. 

1.  The  affective  domain  includes  attitudes,  feelings,  emotions  and  predispositions. 
Attitudes  are  of  paramount  importance  to  learning  and  they  predispose  us  to 
behave  in  certain  ways.  Attitudes  and  values  become  evident  in  behavior. 

2.  Intelligence  is  defined  as  behavior  and  is  skill  based.  It  is  a  dynamic 
characteristic  that  can  be  taught  and  learned.  Intelligence  is  expressed  in 
behavior. 

3.  Tests  and  exams  evaluate  the  acquisition  of  concepts  and  skills,  but  observation 
of  behavior  identifies  the  acquisition  of  attitudes. 

4.  A  student  working  to  potential  is  dynamic  in  that  the  student  keeps  trying, 
continues  to  strive,  and  does  not  give  up.  Potential  is  not  a  preordained  state  to 
be  filled,  it  is  learning  behavior  which  can  be  observed. 

5.  Learning  is  celebrated  when  we  tell  what  the  student  has  learned,  what  is  not  yet 
learned  and  what  plans  we  have  to  do  more.  In  its  ideal  form,  learning  is  a 
joyful  process. 

6.  The  core  of  the  curriculum  is  teaching  behaviors  which  allow  students  to  make 
informed  choices.  The  act  of  choosing  well  requires  a  strong  self-image. 
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Design 


Development 

The  coordinator  first  reviewed  the  literature,  then  set  out  to  discover  what  affective 
learning  was  seen  as  important  to  people  in  Lethbridge.  Personal,  open-ended,  yet 
structured  and  focused,  interviews  were  conducted  with  100  people.  Sampling  was 
purposive  rather  than  random.  Those  interviewed  were  students  from  grades  1  to 
12,  parents,  teachers,  administrators,  trustees,  clergy,  professionals,  post- 
secondary  educators,  employers  and  people  involved  in  correctional  work. 
Responses  from  all  of  the  constituents  were  recorded. 

The  responses  were  sorted,  categorized,  classified,  refined  and  grouped  into  five 
general  areas.  Affective  learning  was  evident  in: 

1.  behaviors  that  showed  self- worth, 

2.  behaviors  that  dealt  with  interpersonal  relationships, 

3.  behaviors  relating  to  world  awareness  and  the  expectations  of  the  work 
world, 

4.  behavior  patterns  of  learning, 

5.  behaviors  relating  to  spiritual  life,  values  and  beliefs  about  purpose  in  life. 


Field  Testing 

One  stage  of  field  testing  was  the  dissemination  and  explanation  of  the  indicators  to 
district  teachers,  parents,  students,  special  interest  groups,  teachers  outside  the 
district,  the  school  board  and  the  media. 

The  indicators  and  behaviors  were  translated  into  French  for  distribution  to  and 
response  from  the  French  immersion  teachers  in  the  district. 

The  coordinator  observed  student  behavior  in  district  classrooms  and  reported  on 
the  many  positive  behaviors  observed.  Teachers  reported  that  the  observation  of 
positive  behavior  had  forced  them  to  look  at  their  class  from  a  different  vantage 
point,  to  look  for  the  positive  rather  than  dwell  on  the  negative,  and  to  be 
encouraged  by  the  positive  behaviors  that  were  now  so  evident. 

As  a  result  of  the  field  test,  it  became  obvious  that  teachers  needed  to  focus  attention 
on  specific  behaviors  to  avoid  general  overviews  that  lacked  detail.  Teachers  needed 
to  observe  positive  rather  than  negative  behavior.  They  needed  to  be  trained  in 
differentiating  between  inference  and  observation.  They  needed  to  see  the  value  of 
observing  individual  students  not  just  the  class  as  a  whole. 

Field  testing  showed  the  following. 

1 .  Indicators  of  affective  learning  need  not  be  totally  comprehensive  but  they  must 
be  typical. 

2.  Reporting  of  observation  need  not  be  of  all  affective  indicators,  for  all  students, 
at  all  times;  but  reporting  of  significant  observations  of  individual  students  is 
helpful. 
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3 .  Reporting  of  affective  learning  is  reporting  of  the  observation  of  positive 
behavior;  but  observation  of  negative  behavior  is  the  basis  for  action  plans  to 
remedy  the  deficiencies. 

4.  Checklists  are  useful  reminders  of  behaviors  observed,  but  they  are  too 
limiting. 

5 .  Published  tests,  instruments  and  measures  of  affective  learning  were  not  seen  to 
be  the  most  valid  and  pure  measure  of  behavior. 

6.  Behavior  is  observed  in  process  and  in  product  by  teachers;  behavior  is 
observed  by  parents  and  peers;  behavior  is  observed  by  students  themselves 
when  given  time  and  freedom  to  reflect 

7 .  In  order  to  operationalize  the  indicators,  there  would  need  to  be  more  specific 
descriptors  of  representative  behaviors  for  each  indicator. 

8 .  There  would  need  to  be  ways  to  report  observations  in  a  positive  manner. 
These  ways  would  need  to  be  easy,  efficient  and  effective  to  use. 

Actions,  typical  of  behaviors  that  teachers  see,  were  collected  from  the  field  tests 
and  from  groups  of  teachers.  These  actions  are  not  prescribed  actions  that  students 
are  required  to  display,  but  rather  they  are  descriptions  of  typical  actions  of  students 
who  show  affective  learning.  These  actions  were  systematized  and  further  refined 
by  collaborating  with  practicing  teachers.  The  descriptions  expanded  horizons  and 
were  included  in  a  folder  of  possible  actions  to  observe  as  evidence  of  the  presence 
of  a  behavior. 


A  Celebration  of  Learning 

Figure  1  presents  a  conceptual  model  of  the  relationship  of  all  five  areas  of  affective 
learning.  The  model  represents  the  formation  and  growth  of  behaviors  that  display 
positive  attitudes  toward  the  SELF  take  place  through  the  interrelated  experiences  of 
dealing  with  OTHERS,  through  a  growing  awareness  of  the  WORLD,  and  through 
the  process  of  LEARNING.  The  SPIRITUAL  dimension  unifies  the  other  four,  by 
identifying  a  purposiveness  in  life,  its  events  and  activities;  it  provides  the  hope  that 
leads  the  learner  on. 


Negative  Schema 

We  validated  the  trustworthiness  of  the  list  of  positive  behaviors  by  testing  it 
against  a  mirror  image  of  negative  behaviors  as  is  shown  in  Figures  2  and  3. 
Teachers  were  consulted  to  validate  that  the  negative  behaviors  were  those  which 
show  affective  decay  rather  than  affective  growth.  Negative  behaviors  were 
validated  also  from  interview  responses  of  behaviors  which  showed  a  lack  of 
learning.  We  validated  the  behaviors  in  a  practical  sense  in  the  classrooms  of 
practicing  teachers.  We  asked  teachers  to  add  to  or  delete  behaviors  which  were  not 
relevant  to  teaching  and  learning.  The  coordinator  observed  classrooms  to  validate 
the  behaviors.  The  responses  of  teachers  were  used  and  the  list  of  behaviors  was 
refined,  expanded,  defined  in  more  precise  terms  and  validated  again  in  the 
classroom. 
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Figure  1:  The  Conceptual  Model 


Signs  of  Affective  Learning 

The  indicators  of  learning  are  Self  Worth,  Relating  to  Others,  World  Awareness, 
Learning,  and  Spiritual  Life.  The  behaviors  which  are  observed  as  evidence  of 
these  indicators  follow. 


in  SELF  WORTH 

1 .  Demonstrates  confidence,  self- worth,  and  security  by  taking  risks,  accepting 
error  with  ease  and  good  humor,  maintaining  a  healthy  life  and  taking  pride  in 
posture  and  appearance. 

2.  Assumes  responsibility  by  accepting  the  consequences  of  actions,  accepting 
leadership,  cooperating  with  responsible  leaders,  and  demonstrating  emotional 
well-being. 

3.  Perseveres  in  developing  gifts  and  talents  through  consistent  effort,  efficient  use 
of  time,  and  taking  pride  in  accomplishments. 
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in  RELATIONSHIP  WITH  OTHERS 

4.  Shows  consideration  and  respect  for  authority  while  understanding  the 
legitimacy  of  dissent  by  choosing  appropriate  ways  to  agree  or  disagree,  by 
responding  to  adult  direction  and  by  following  rules. 

5.  Appreciates  the  accomplishments  of  others  by  affirming  and  encouraging  them, 
being  sensitive  to  their  feelings,  and  by  accepting  winning  or  losing  in  a 
gracious  manner. 

6.  Respects  the  rights,  ideas  and  property  of  others  by  valuing  diversity  of  ideas, 
caring  for  the  property  of  others,  accepting  and  understanding  differences,  and 
respecting  the  right  of  others  to  learn. 

in  WORLD  AWARENESS 

7.  Displays  a  belief  in  the  value  of  citizenship  by  contributing  to  the  building  of 
community,  participating,  sharing  information  or  time  or  energy  or  resources, 
cooperating  in  groups,  volunteering  and  becoming  involved  in  community 
endeavors. 

8.  Demonstrates  a  commitment  to  social  justice  and  fair  play  by  preserving  the 
environment,  conserving  natural  resources,  using  equipment  and  materials 
responsibly,  and  sharing  generously. 

9.  Exemplifies  dependability  and  dedication  by  being  present,  punctual,  prepared, 
attentive,  diligent,  good  humored  and  able  to  work  independently. 

in  LEARNING 

10.  Shows  a  desire  to  learn  and  to  search  for  truth  by  being  open  and  receptive  to 
learning,  affirming  it  and  reflecting  it  with  happiness,  enthusiasm  and  satisfaction. 

11.  Demonstrates  self-direction  and  self-motivation  by  initiating  activities, 
accepting  challenges,  extending  knowledge,  exploring  possibilities,  expanding 
horizons  and  creating. 

12.  Displays  intellectual  curiosity  by  maintaining  an  open  and  inquisitive  mind, 
being  flexible  to  change,  using  skills  of  intelligence  such  as  critical  thinking 
skills,  asking  informed  questions,  and  employing  a  variety  of  problem-solving 
techniques. 

in  SPIRITUAL  LIFE 

13.  Demonstrates  faith  in  a  Supreme  Being  and  in  a  system  of  values  which 
distinguishes  right  from  wrong  by  displaying  truth,  honesty,  integrity,  justice, 
and  inner  peace. 

14.  Exemplifies  hope,  trust  and  purpose  in  life  by  being  prudent  and  wise  in 
decision  making,  and  working  toward  a  goal. 

15.  Personifies  love  by  showing  acceptance,  thankfulness,  tolerance,  compassion, 
reconciliation,  and  a  sense  of  selflessness  through  considerate,  responsible  and 
trustworthy  behavior. 
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Implementation 

The  coordinator  was  actively  involved  in  the  implementation.  She  communicated 
the  project  and  findings  to  district  teachers  formally  on  a  school-by-school  basis 
and  informally  on  a  one-to-one  basis  in  staff  rooms  and  classrooms.  The 
coordinator  was  seen  as  a  disinterested  participant  and  teachers  were  free  to  discuss 
the  negative,  the  positive,  the  interesting  and  the  not  so  interesting  results  of 
introducing  the  concept  of  observing  behavior  as  an  authentic  assessment  tool. 

The  demands  of  the  classroom  present  some  difficulty  in  gaining  access  to  teachers' 
time.  The  coordinator,  a  teacher,  overcame  this  problem  at  one  stage  of  the 
implementation  by  taking  over  the  teaching  and  evaluating  role  of  classrooms  so 
that  teachers  could  experience  the  benefits  of  observing  the  positive  behaviors  of 
their  students  in  authentic  situations.  At  another  stage,  she  observed  the  behavior 
of  students  in  classrooms  and  reported  orally  to  the  students  and  the  teacher  the  type 
of  learning  behaviors  that  were  observed.  This  exercise  assisted  teachers  in  looking 
for  positive  behaviors  rather  than  seeing  only  negative  behaviors  which  seem  to 
jump  out  at  you. 

The  concept  of  celebrating  learning  by  observing  behavior  must  fit  into  an  already 
existing  system  of  assessment,  evaluation  and  communication.  To  facilitate  this, 
the  coordinator  became  a  participant  in  school  based  discussion,  planning  and 
redesigning  of  reporting  systems.  Since  each  of  the  eight  schools  in  the  district 
has  its  own  reporting  system,  the  coordinator  incorporated  the  results  of  the 
project  in  a  different  manner  for  each  group. 

Implementation  required  making  distinctions  between  observing  and  inferring. 
The  superintendent,  who  regularly  communicates  concepts,  ideas,  theories  and 
assumptions  to  all  teachers  in  the  system  through  "Viewpoints"  newsletters, 
exemplified  observation  and  inference  in  one  of  the  newsletters. 

During  the  implementation  process  it  became  evident  that  at  the  secondary  level 
some  teachers  viewed  affective  learning  as  an  add-on  to  the  prescribed  curriculum. 
They  did  not  feel  that  there  was  time  for  anything  extra  in  a  semester.  To  answer 
this  objection,  the  coordinator  explained  through  personal  contacts  with  both  the 
junior  high  school  and  the  senior  high  school  that  the  affective  domain  is 
interwoven,  interrelated  and  intertwined  with  every  other  domain  to  the  extent 
that  it  is  impossible  to  refrain  from  addressing  affect 

To  assist  the  junior  and  senior  high  schools  in  implementing  the  reporting  of 
positive  behaviors  apart  from  the  cognitive  and  skill  areas,  the  coordinator 
systematized  the  computer  comments  used  on  student  progress  reports.  Comments 
were  organized  into  administrative,  achievement,  skill  and  behavior  sections.  The 
behavior  comments  were  divided  into  those  which  described  positive  actions 
observed  and  those  which  described  negative  actions  observed.  An  attempt 
was  made  to  delete  inferences  wherever  possible. 

There  was  limited  acceptance  of  the  idea  in  the  high  school  but  much  greater 
acceptance  in  the  junior  high  school.  However,  some  teachers  in  both  junior  and 
senior  high  schools  adopted  the  concept  of  affimiing  positive  behaviors  with  good 
news  notes,  rewriting  the  computer  comments,  and  teaching  behaviors  in  classes 
such  as  Career  and  Life  Management,  Religious  Studies,  and  Health  and  Personal 
Development.  We  discontinued  an  attempt  to  describe  growth  in  behavior  that 
would  differentiate  between  the  nuances  of  behavior  as  the  students  mature.  The 
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vast  array  of  possibilities  would  be  extremely  cumbersome  and  would  violate  our 
criteria  of  practicality  and  simplicity.  Since  behavior  does  not  develop  in  a  linear 
fashion,  we  found  it  of  no  value  to  describe  behavior  as  a  series  of  benchmarks  or 
standards  representative  of  specific  age  groups. 

No  portion  of  the  project  was  mandated  to  teachers  in  Lethbridge  Catholic  Schools. 
It  was  offered  to  teachers  to  accept  or  reject  in  part  or  in  whole.  The  only  thing 
asked  was  that  teachers  continue  to  give  feedback  to  the  coordinator  on  how  they 
used  the  behavior  indicators,  how  they  observed  behavior,  what  instruments 
worked,  what  ideas  did  not  work,  what  reactions  they  received  from  parents 
and  students,  and  any  suggestions  they  had  to  offer. 

The  cardinal  rule  in  implementing  the  project  hinged  on  being  able  to  show  an 
immediate  relationship  between  the  theoretical  and  the  practical  aspects  of  it  The 
tension  between  the  practical  and  the  theoretical  was  lessened  when  it  was  made 
clear  that  the  theory  could  be  put  into  practice  in  a  simple  and  efficient  manner  that 
would  prove  beneficial  to  teachers,  students  and  parents. 


Instruments 

Research  demonstrated  that  any  procedure  for  observation  of  behaviors  must  meet 
several  criteria.  The  procedure  must  be  simple;  the  behaviors  must  be  systematized 
and  observable;  the  recording  of  observations  must  be  time  efficient;  and  the 
reporting  of  behaviors  observed  must  be  useful  to  teachers  and  helpful  to  students 
and  parents. 

The  instruments  designed  are  simple;  the  procedure  is  simple  and  time  efficient; 
the  indicators  are  defensible.  Observation  can  be  used  in  the  classroom,  on  the 
playground  and  at  home.  Behaviors  are  observable  by  parents,  teachers,  and 
students.  Instruments  for  recording  observations  are  open-ended.  Instruments 
are  designed  to  record  and  report  positive  behaviors  in  a  constructive  manner. 
Recording  of  observations  is  simple;  reporting  is  simple  yet  affirming  to  students, 
informative  for  parents  and  useful  for  teachers. 

Each  of  the  instruments  developed  is  described  below. 

1 .  A  brochure  to  identify  the  15  indicators  of  growth  in  behaviors.  The  behaviors 
are  illustrated  and  displayed  on  five  separate  pages  which  identify  them  in  chart 
form  on  graduated  page  lengths.  The  artist  depicts  birds  displaying  bird 
behaviors  as  they  grow  to  maturity  and  become  independent. 

2.  A  folder  to  identify  seven  representative  actions  for  each  of  the  15  behaviors. 
The  folder  of  105  specific  behaviors  helps  to  focus  observation. 

3 .  A  class  checklist  of  the  15  indicators  to  allow  for  a  quick  method  of  recording 
reminders  of  observations.  Later,  the  observations  can  be  written  in  journals 
or  anecdotes. 

4.  Self-evaluations  for  students.  There  are  separate  versions  for  primary  (grades 
1-3),  elementary  (grades  4-6)  and  secondary  (grades  7-12)  students.  The 
primary  student  self-evaluations  are  five  individual  sheets  covering  each  area  of 
behavior.  Each  sheet  asks  eight  questions  to  help  students  think  about  how  they 
behave  and  to  record  their  thoughts  by  coloring  happy  faces.  The  elementary 
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self-evaluation  consists  of  one  sheet  that  poses  seven  questions  for  each  of  the 
five  areas.  It  also  presents  an  opportunity  to  reflect  on  which  behaviors  they  do 
well,  which  behaviors  have  improved  and  which  ones  still  need  improvement 
The  secondary  self-evaluation  is  similar  to  the  elementary  one  but  the 
vocabulary  and  reflections  take  into  account  the  maturity  of  the  students. 

5 .  Good  News  Notes  for  students  and  parents.  The  notes  are  designed  to  allow 
teachers  to  give  immediate  feedback  to  students  and  parents  when  constructive 
and  encouraging  behavior  has  been  observed  at  school.  The  carbon  copy  of  the 
report  allows  teachers  to  keep  a  record  of  the  observation  for  future  assessment, 
discussion  or  conference. 

It  may  be  of  interest  to  note  that  the  first  attempt  at  preparing  good  news  note 
pads  met  with  less  than  encouraging  results.  The  blank  note  pads  were  being 
used  as  just  that  -  note  pads.  The  redesigned  note  pads  have  a  carbon  copy,  are 
addressed  to  student  and  parents,  have  part  of  the  message  printed  on  the  note, 
and  provide  space  for  teachers  to  describe  the  behavior  observed,  the  date  and 
their  signature. 

6.  A  presentation  kit  to  enclose  the  instruments.  The  kit  briefly  describes  how  to 
use  the  kit,  what  assumptions  are  made,  and  the  concept  of  the  celebration  of 
learning. 


Observations 

The  purpose  of  the  Lethbridge  Catholic  EQI  Project  was  to  find  signs  of  learning  in 
the  affective  domain.  The  project  began  by  identifying  indicators  of  the  formation 
and  presence  of  those  attitudes  deemed  to  be  important  goals  of  education.  The 
project  set  out  to  design  a  system  to  monitor,  record,  and  report  behavior,  the 
expression  of  attitudes.  Unexpected  purposes  of  the  project  became  obvious;  one 
was  the  need  to  train  educators  to  observe  positive  behavior,  another  was  the  need 
to  assist  educators  to  teach  and  to  model  behaviors. 

We  found  specific  observable  behaviors  indicating  affective  growth,  that  are  valued 
by  their  stakeholders,  promoted  by  Lethbridge  Catholic  School  policies,  mandated 
by  Alberta  Education,  and  supported  in  the  literature. 

Some  observations  of  the  project  follow. 

1 .  There  is  significant  similarity  between  the  responses  of  the  various  stakeholders 
to  the  interview  questions.  Stakeholders  were  asked  to  identify  affective 
qualities  which  they  valued  as  significant  outcomes  of  schooling.  The  similarity 
of  responses  resulted  in  the  indicators  of  affective  learning  identified  by  this 
project. 

2 .  The  behaviors  identified  as  indicators  of  affective  learning  are: 

•  acceptable  -  They  have  not  been  disputed. 

•  valid  -  They  have  shown  trustworthiness. 

•  useful  -  They  are  practical  in  the  classroom. 

•  transferable  -  Other  school  districts  have  adopted  them. 

•  comprehensive  -  There  have  been  no  further  suggestions. 

•  stimulating  to  teachers  in  their  practice. 
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3 .  In  the  implementation  of  the  project,  it  is  important  to  note  that  we  did  not 
mandate  the  use  of  the  concept  or  the  instruments.  We  offered  them  to  teachers 
to  use  in  the  existing  system  to  the  degree  that  teachers  found  them  useful.  Our 
observation  is  that  many  teachers  are  ready  to  use  or  are  using  the  behaviors  and 
their  representative  actions  to  assess  affective  learning.  They  use  the 
instruments: 

•  to  teach  behavior 

•  to  describe  behavior 

•  to  talk  to  parents  and  assist  them  in  parenting 

•  to  use  as  a  common  reference  in  conferences 

•  to  help  write  anecdotal  reports 

•  to  get  ideas  for  report  card  comments 

•  to  set  objectives  and  goals  for  behavior 

•  to  plan  lessons  and  activities 

•  to  focus  their  observations  on  specific  actions 

•  to  evaluate  the  affective  areas  of  learning 

•  to  describe  growth  and  development  of  students  in  special  education 
programs, 

4.  In  the  past  we  lacked  an  adequate  vocabulary  to  simultaneously  identify 
behavioral  growth  and  deficiency.  We  recognize  the  need  to  identify  the 
deficiency  but  not  destroy  the  child.  We  have  difficulty  reporting  that  a  student 
needs  further  development  in  behavior.  The  long  list  of  behaviors  prepares  us  to 
accept  that  behavior  is  not  perfect  in  anyone,  but  that  we  can  see  growth  in 
behavior  if  we  choose  to  do  so. 

5 .  This  project  used  observation  of  process  and  of  product  to  assess  behavior  in  an 
attempt  to  avoid  the  cognitive  component  of  instrumental  approaches  and  to 
provide  a  more  pure  measure  of  affective  learning.  The  professional  observation 
of  teachers  can  be  augmented  by  observations  of  parents,  students  and  peers. 


Implications 

1 .  Behavior  is  dynamic  and  does  not  follow  a  linear  pattern  of  development 
Attempts  to  set  standards  by  which  behavior  can  be  rated,  graded  or  quantified 
would  serve  little  purpose. 

2.  The  behavior  of  students  is  observed  in  places  other  than  the  school.  It  is 
important  that  multiple  observers  contribute  to  the  assessment  of  affective 
learning. 

3 .  The  procedure  of  using  lists  of  descriptors  to  identify  affective  learning  is 
instructive  for  students  and  parents.  It  is  beneficial  to  teachers  who  can  then 
base  instructional  and  reporting  decisions  on  direct  observation. 

4 .  Including  observation  of  behavior  as  authentic  assessment  of  learning  broadens 
the  base  of  assessment  and  gives  hope  to  students  who  have  traditionally  been 
assessed  only  on  cognitive  achievement  and  skill  development. 

5 .  Good  News  Notes  give  immediate  feedback  to  students  on  their  behaviors 
which  are  conducive  to  learning. 
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6.  The  precision  of  the  behaviors  strengthens  their  use  in  observation  and  reduces 
the  need  for  inference. 

7 .  There  is  resistance  to  implementing  something  new  into  an  existing  system. 
The  new  concept  requires  that  old  concepts  change  in  shape  or  in  significance. 
It  requires  flexibility  in  the  system.  In  order  for  a  new  concept  to  be  accepted 
and  incorporated  it  must  not  only  be  simple,  efficient,  effective  and  beneficial, 
but  it  must  be  seen  to  be  so. 


Recommendations 

For  Lethbridge  Catholic  Schools 

1 .  That  junior  and  senior  high  schools  incorporate  descriptors  of  the  Signs  of 
Affective  Learning  in  their  computerized  comments  for  reporting  student 
progress. 

2.  That  teachers  acquaint  themselves  with  the  descriptors  of  affective  learning 
presented  in  the  Celebration  of  Learning  kit. 

3 .  That  teachers  make  use  of  the  Good  News  Notes,  provided  in  the  kit,  to  affirm 
student  behavior. 

4 .  That  teachers  make  precise  distinctions  between  observed  student  behavior  and 
inferences  based  on  observations. 

5 .  That  teachers  review  the  concept  of  A  Celebration  of  Learning  at  student- 
parent-teacher  interviews. 

6 .  That  the  district  adopt  the  concept  of  A  Celebration  of  Learning  and  the 
indicators  of  learning  as  the  basis  for  describing  student  progress  in  the 
affective  domain. 


For  Other  Districts 

7 .    That  other  districts  evaluate  the  celebration  of  learning  concept  and  supporting 
materials  to  determine  their  relevance  to  local  needs. 


For  Alberta  Education 

8 .  That  Alberta  Education  adopt  the  concept  of  A  Celebration  of  Learning  and  the 
indicators  of  learning  as  the  basis  for  describing  student  progress  in  the 
affective  domain. 

9 .  That  the  department  investigate  the  application  of  the  concept  of  A  Celebration 
of  Learning  to  other  domains. 
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10.  That  it  adopt  the  definition  of  a  student  "working  to  potential"  as  a  student 
who  keeps  trying  and  does  not  give  up.  Potential  is  a  dynamic,  continuing 
condition  not  a  state  to  be  realized.  It  follows  that  the  criterion  for  checking 
any  feedback  on  student  performance  is  whether  it  allows  and  encourages 
students  to  keep  trying.  The  skill  of  making  choices  becomes  central  for 
students. 


Follow-up 

A  presentation  kit  including  all  relevant  materials  is  being  disseminated  to  the 
teachers  of  Lethbridge  Catholic  Schools. 

The  research  results  are  being  presented  to  interests  groups  on  request. 

We  are  presenting  the  results  of  the  research  project  to  the  University  of  Lethbridge 
to  be  considered  for  possible  use  in  their  teacher  training  program. 

We  are  sending  a  copy  of  A  Celebration  of  Learning  to  the  Student  Evaluation 
Branch  of  Alberta  Education  for  their  consideration. 


Conclusions 

We  feel  confident  that  the  behaviors  we  identified  as  signs  of  learning  in  the 
affective  domain  make  sense.  They  are  research  based  and  they  are  valued 
by  the  community. 

It  is  difficult  to  break  the  cultural  bias  that  wants  to  compare.  Some  people  still  see 
rating,  grading  and  comparing  as  necessary.  Some  people  regard  good  grades  as 
rewards  and  poor  grades  as  goads  to  do  better.  Others  believe  there  is  a  need  for 
children  to  learn  to  fail  and  that  mistakes  are  flaws  which  when  repeated  are  the 
result  of  willfulness.  Yet,  we  propose  that  the  presence  or  absence  of  the  identified 
positive  behaviors  is  significant  in  itself. 

We  find  often  that  schools  are  applying  evaluation  concepts  derived  from  assessing 
the  cognitive  and  the  psychomotor  domains  to  assessment  in  the  affective  domain. 
This  project  has  provided  concepts  and  vocabulary  specific  to  assessing  affective 
learning. 

Negative  behaviors  are  easily  identified  and  are  developed  without  assistance. 
Positive  behaviors  are  not  so  easily  identified  because  we  assume  them  and  they 
are  subject  to  being  overlooked.  They  are  not  always  acknowledged  or  reported 
as  signs  of  learning.  There  appeared  to  be  no  benefit  in  developing  the  negative  side 
of  the  behavior  chart  any  further  than  to  use  it  to  validate  our  findings.  However, 
developing  the  positive  side  produced  objectives  for  student  outcomes  and 
pedagogical  consequences  in  teaching,  demonstrating  and  modeling  behaviors. 
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We  experienced  much  interest  in  the  project  and  anticipation  of  results  for 
implementation.  To  fulfill  the  expectations  of  those  who  collaborated  in  the 
project,  the  results  will  be  disseminated  in  a  presentation  kit  which  will  include 
a  brief  description  of  the  theory,  a  full  description  of  the  identified  behaviors, 
master  copies  of  the  instruments  developed,  support  for  teaching  and  modeling 
behavior,  and  a  user's  guide. 

The  findings  of  this  project  have  a  counter  cultural  quality  in  that  they  go  against 
an  educational  concept  derived  perhaps  from  a  confused  religious  concept.  That 
confusion  equated  sin  and  error.  Behaviors  which  were  in  error  were  sought  out, 
exposed  and  expunged.  But  A  Celebration  of  Learning  would  have  the  teacher 
look  for  opportunities  to  observe  behavior,  to  find  joy  in  the  behavior  of  children 
in  the  act  of  learning,  and  to  take  pleasure  in  letting  them  know  of  it 
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EQI  Accomplishments  and  Next  Steps 


This  final  chapter  of  Achieving  Quality  describes  the  major  accomplishments  of 
the  EQI  initiative,  proposes  potential  provincial  indicators,  and  outlines  some 
next  steps  in  expanding  the  adoption  of  educational  quality  indicators  in  school 
jurisdictions  in  Alberta. 


Accomplishments 

The  Educational  Quality  Indicators  initiative  accomplished  what  it  set  out  to  do 
more  than  three  years  ago.  Each  of  the  past  three  years  unfolded  as  originally 
conceived  -  an  initial  developmental  year  followed  by  field  testing,  and  then  a 
year  to  reflect  upon  the  findings  and  refinement  of  the  indicator  systems.  What 
has  been  learned  can  now  be  shared  with  others  in  Alberta  and  elsewhere. 

The  major  accomplishments  of  EQI  include  the  development  of  enhanced 
indicator  systems  based  on  theory  and  practical  experience;  development  of 
better  ways  to  track  educational  improvement  in  Alberta;  the  collaboration 
between  Alberta  Education  and  the  twelve  school  jurisdictions  to  develop 
indicator  systems;  and  the  involvement  by  the  Conference  of  Alberta  School 
Superintendents  (CASS)  in  organizing  the  EQI  conference. 


A  Direction  for  Improvement 

The  indicator  systems  developed  by  the  participating  school  jurisdictions  are 
based  on  theory  and  practice.  An  extensive  review  of  the  literature  during  the 
developmental  phase  ensured  that  teams  were  aware  of  conceptual  frameworks 
and  incorporated  these  into  developing  their  indicator  systems.  The  field  tests 
tempered  theory  with  practical  realities  to  ensure  that  indicators  that  survived 
this  phase  were  practical  and  had  direct  application  for  school  districts. 

This  direction  for  improvement  incorporated  a  focus  on  student  outcomes  based 
on  the  goals  of  schooling,  a  balance  among  cognitive,  affective  and  behavioral 
outcomes,  multiple  ways  to  collect  and  report  information,  and  active  involvement 
of  educational  partners  (students,  staff,  parents,  stakeholders,  and  the  public). 

The  local  indicator  systems  have  support  within  their  communities  because  of 
the  active  involvement  of  partners  in  their  development.  Several  models  for 
assessing  the  effectiveness  and  efficiency  of  instructional  programs,  delivery 
systems,  and  overall  system  performance  were  developed.  In  addition,  several 
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innovative  alternative  student  assessment  strategies  were  developed.  These 
"products"  can  now  be  adapted  by  other  districts  and  used  to  suit  their  local 
purposes. 

Products 

The  project  teams  have  developed  many  new  approaches  to  assessing  student 
learning  and  evaluating  school  systems,  schools,  programs  and  teachers.  They  are 
valuable  additions  to  the  existing  set  of  measurement  instruments  because  they  are 
based  on  theory  and  refined  through  practice.  Other  districts  can  use  these 
products  to  assess  a  broader  range  of  outcomes.  The  major  products  developed 
are  identified  below. 

•  The  Lacombe/Rocky  Mountain  Project  developed  a  comprehensive  school 
system  review  process  that  can  be  adapted  by  other  school  jurisdictions.  The 
data  for  student  learning  in  intellectual,  social,  personal,  and  vocational  areas 
can  serve  as  points  of  reference  for  other  districts  wishing  to  measure  these 
areas.  The  Handbook  of  Procedures  for  Implementing  the  School  System 
Review  Process  is  a  complete  guide  for  implementing  the  review. 

•  The  Grande  Prairie  Project  developed  prototype  school  and  district  profiles. 
The  profiles  report  indicators  of  student  achievement,  school  climate,  quality 
of  instruction,  and  funding.  The  district  is  working  on  a  computer  system  to 
facilitate  production  of  the  profiles. 

•  The  Lethbridge  Public  Project  developed  a  collaborative  model  for  school  and 
program  evaluation  that  can  be  generalized  to  other  programs  and  adapted 

by  other  districts. 

•  The  Spirit  River  Project  identified  26  behaviors  of  effective  teaching  and 
grouped  them  in  four  areas  -  planning  and  preparation,  instructional  strategies, 
communication  skills,  and  pupil/teacher  relationships.  Teachers  were 
observed  in  1991  and  1992  and  composite  performance  profiles  prepared 

for  both  years.  The  results  serve  to  identify  jurisdiction-level  professional 
development  needs.  Other  districts  could  adapt  the  procedures  for  local  use. 

•  The  Fort  McMurray  Catholic  Project  incorporated  the  principles  of  outcome- 
based  education  (OBE)  into  grades  3  to  10  mathematics  classes.  OBE  requires 
students  to  demonstrate  mastery  of  the  significant  outcomes  at  each  level. 
Alberta  Education  has  released  Every  Child  Can  Learn  which  describes  the 
philosophy  and  principles  of  OBE,  and  can  serve  as  a  collaborative  guide  for 
jurisdictions  interested  in  this  process.  Fort  McMurray  Catholic  and  Alberta 
Education  are  planning  a  Canadian  OBE  conference  to  be  held  at  the 
Fantasyland  Hotel,  May  17-19, 1993. 

•  The  Edmonton  Public  Project  developed  mechanisms  for  student  portfolio 
assessment,  the  use  of  computer  technology  for  communicating  and  storing 
information  about  student  growth,  and  established  and  validated  curriculum- 
based  district  level  assessment  strategies. 
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•  The  Calgary  Public  and  Catholic  Project  developed  indicators  of  quality 
student  performance  in  art  and  mathematics.  The  EOI-Art  Handbook  and 
Problem  Solving  Profiles  provide  exemplars  of  student  performance  in  art 
and  mathematics  across  elementary,  junior  and  high  school  levels. 

•  The  Brooks  Project  explored  ways  to  measure  responsible  student  behavior. 
The  project  collected  data  from  a  variety  of  sources  and  involved  the 
community  in  reaffirming  its  commitment  to  positive  student  behavior. 
Brooks'  Character  in  Our  Community  Beliefs  incorporate  these  beliefs 
about  responsible  student  behavior. 

•  The  Fort  McMurray  Public  Project  developed  a  three-level  approach  to 
measuring  student  social  outcomes.  The  Social  Skills  Diagnostic  Screen  is  a 
new  instrument  measuring  social  development  in  personal,  interpersonal, 
social  reasoning  and  classroom  behavior.  Products  include  an  administration 
manual  and  a  manual  of  activities  to  help  students  develop  effective  social 
skills. 

•  The  Lethbridge  Catholic  Project  developed  A  Celebration  of  Learning  which 
includes  five  indicators  of  affective  behavior  -  self-worth,  interpersonal 
relationships,  world  awareness,  learning  and  spiritual  life.  Products  include 

a  presentation  kit  which  contains  a  brochure,  a  class  checklist,  self-evaluation 
sheets  for  students,  and  Good  News  Notes. 

The  results  for  selected  indicators  provide  a  point  of  reference  for  other  districts 
that  may  wish  to  collect  this  type  of  information.  For  example,  in  addition  to 
provincial  achievement  and  diploma  results,  the  School  Subjects  Attitude  Scales 
were  used  by  both  the  Lacombe/  Rocky  Mountain  and  Fort  McMurray  Catholic 
projects  to  provide  a  measure  of  student  attitudes.  Participation  rates  in  Alberta 
achievement  tests  and  diploma  exams  are  reported  in  the  Lacombe/Rocky 
Mountain  and  Brooks  summaries;  these  serve  as  a  complementary  indicator 
to  achievement. 


Collaboration 

The  EQI  initiative  has  become  a  model  for  collaboration  between  government  and 
school  jurisdictions.  Despite  changes  in  project  coordinators,  superintendents, 
and  three  years  of  change  in  Alberta,  the  initiative  met  its  target  and  was 
completed  on  time. 

The  January  1993  EQI  Conference  marked  the  formal  conclusion  of  the  initiative, 
but  the  beginning  of  a  new  era  of  cooperation  among  educational  partners 
interested  in  improving  education.  Organized  by  the  Conference  of  Alberta 
School  Superintendents,  the  conference  provided  an  opportunity  for  the  ten 
project  teams  to  present  the  materials  they  developed  and  the  results  achieved 
over  the  past  three  years.  More  than  300  delegates  from  British  Columbia 
to  Newfoundland  attended  the  conference.  This  was  a  major  accomplishment 
because  it  was  the  first  time  that  Alberta  Education  held  a  conference  on  a 
research  endeavor,  and  the  first  time  that  CASS  organized  such  an  event  for  an 
Alberta  Education  initiative.  The  precedent  bodes  well  for  future  cooperative 
projects  between  the  department  and  CASS. 
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Communication 

EQI  partners  participated  in  symposia  at  two  meetings  of  the  Canadian  Society 
for  the  Study  of  Education,  one  of  the  Learned  Societies  of  Canada.  In  1990,  five 
project  teams  participated  in  the  first  EQI  symposium  in  Victoria,  BC  at  which 
they  described  their  proposed  indicator  systems.  In  1992,  seven  teams 
participated  in  the  second  symposium  in  Charlottetown,  PEI  where  they  shared 
preliminary  results.  Both  were  well  attended  and  the  feedback  from  the 
delegates  was  very  positive.  The  proceedings  of  both  symposia  are  available. 
The  symposia  helped  participants  clarify  their  projects  and  inform  the  Canadian 
academic  community  of  the  initiative. 

In  addition  to  these  two  national  symposia,  teams  participated  in  local  workshops, 
professional  development  days,  teachers'  conventions  and  conferences,  and  will 
continue  to  do  so.  Some  also  took  part  in  national  and  international  conferences. 
Requests  for  information  about  EQI  have  come  from  governments,  school 
jurisdictions,  universities  and  other  organizations  across  Canada,  the  United 
States,  Europe,  Asia  and  Australia. 


Potential  Provincial  Indicators 

A  major  purpose  of  the  EQI  initiative  was  to  develop  a  set  of  indicators  for  a 
provincial  indicator  system.  At  the  fifth  provincial  meeting  in  January  1992,  EQI 
partners  discussed  potential  provincial  indicators,  effective  practices,  follow-up 
plans  and  issues.  Participants  recommended  that  provincial  indicators  should: 

•  focus  on  student  outcomes 

•  be  based  on  the  goals  of  schooling  and  desirable  personal  characteristics 

•  represent  a  balance  among  cognitive,  affective  and  behavioral  outcomes 

•  provide  corollary  evidence  (tests,  surveys,  interviews,  records) 

•  provide  multiple  perspectives  (student,  staff,  parents,  stakeholders,  the  public) 

Table  1  presents  the  indicators,  measures  and  sources  for  the  student  outcomes  for 
which  there  was  consensus.  Each  indicator  is  tied  to  one  or  more  of  the  goals  of 
schooling. 

The  value  of  the  proposed  indicators  is  twofold.  Alberta's  achievement  testing 
and  diploma  examination  programs  already  exist  and  provide  districts  with  annual 
results  for  their  students.  Adding  indicators  of  participation  and  completion 
provide  a  more  comprehensive  picture  of  student  success.  Whereas  achievement 
provides  a  measure  of  academic  excellence,  participation  provides  an  indication  of 
equity.  Together  they  provide  a  better  picture  of  performance  than  either  alone 
does.  Moreover,  evidence  of  exemplary  products  by  students  kept  in  a  portfolio 
provides  a  measure  of  the  diversity  of  student  performance.  The  proposed 
indicators  can,  therefore,  not  only  be  aggregated  and  reported  at  the  provincial 
level,  but  also  at  the  district  and  school  levels.  Those  involving  student  diversity 
and  nonacademic  areas  can  be  collected  and  reported  locally.  Jurisdictions  then 
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Table  1 

Potential  Provincial  Indicators 


Goals1  Indicator 


Measures 


Source(s) 


Students: 

1,2,3  achievement 


1.2 


1-7 


1,2 


1-7 


4-7 


5,6 


participation 


1,2,7  creativity 


graduation 


attitudes 


self-esteem 


satisfaction 


behavior 


responsibility 


fitness 


grades  3, 6, 9  achievement  tests 
grade  12  diploma  examinations 
Rutherford  Scholarships 
performance  assessment 

percentage  of  students  taking 
courses  and  writing  tests/exams 

evidence  of  exemplary  work  kept  in 
portfolios  (student  products,  videos) 

percentage  of  students  who  graduate 
from  high  school 

School  Subjects  Attitude  Scales  2 
other  attitude  surveys 

available  instruments  3 
student  surveys 
observation 

satisfaction  surveys 
personal  interviews 

A  Celebration  of  Learning  4 
Social  Skills  Diagnostic  Screen  $ 

homework  completion 

attendance 

vandalism 

Canada  Fitness  Award 


students 

district 

students 

district 

students 

students 

teachers,  parents 
students,  teachers,  public 

students,  teachers,  parents 


students 

district 

district 

students 


1  Goals  of  schooling:  1  communication,  2  content,  3  skills,  4  well-being,  5  citizenship,  6  world  of  work, 
7  desirable  personal  characteristics.  Goals  1  and  3  are  cognitive  in  nature,  goal  2  contains  both  cognitive 
and  affective  aspects,  and  goals  4  to  7  essentially  encompass  all  three  domains. 

2  Students'  attitudes  toward  subjects  are  assessed  on  three  factors:  evaluation,  usefulness,  difficulty. 

3  There  are  several  instruments  available.  Consult  the  EQI  Inventory  of  Assessment  Instruments. 

4  Student  behaviors  are  described  in  five  groups  which  show  learning  in:  self-worth,  relating  to  others, 
world  awareness,  learning,  spiritual  life.  See  the  Lethbridge  Catholic  report 

5  Student  behaviors  are  categorized  into  four  areas:  personal  factors,  interpersonal  factors,  social 
reasoning  skills,  classroom  behaviors.  See  the  Fort  McMurray  Public  report. 
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are  free  to  tailor  their  information  reports  to  their  publics  based  on  local  priorities 
and  expectations.  Therefore,  accountability  can  be  served  both  provincially  and 
locally,  while  respecting  local  autonomy. 

Measuring  indicators  in  multiple  ways  gives  districts  confidence  in  the 
information  provided  by  a  variety  of  sources.  Provincial  sources  provide  common 
indicators  which  ensure  comparability  among  districts  and  provide  an  overall 
picture  for  the  province.  For  example,  when  achievement  is  measured  by 
achievement  tests  and  diploma  exams,  Rutherford  Scholarships,  and  performance 
assessments,  districts  can  see  how  well  students  at  different  levels  (grades  3,  6, 9, 
12)  compare  to  the  provincial  average.  Trends  over  time  indicate  whether 
improvement  is  occurring. 

Many  of  the  products  described  in  the  previous  section  can  be  used  by  other 
districts  immediately.  For  example,  the  Social  Skills  Diagnostic  Screen  can 
measure  student  social  skills  and  A  Celebration  of  Learning  describes  student 
affective  behavior.  Schools  are  microcosms  of  society,  and  social  and  affective 
skills  are  important  areas  for  student  growth  and  development. 

The  project  teams  developed  a  common  understanding  of  the  implications  of 
better  information.  This  can  now  be  shared  with  others  to  help  them  in 
planning,  setting  policies  and  practices,  and  making  decisions. 


Next  Steps 

The  final  project  reports  will  be  published  as  they  are  finalized.  They  will  contain 
full  documentation  of  the  indicators,  methods  and  findings  of  the  individual 
projects.  Alberta  Education  will  release  each  report  as  it  becomes  available  in 
the  next  few  months. 

Alberta  Education  will  also  produce  a  synopsis  of  the  EQI  initiative  for  wide 
distribution  to  people  interested  in  the  handbooks  and  instruments  that  have  been 
produced.  These  will  be  made  available  on  a  cost-recovery  basis  through  the 
Learning  Resources  Distributing  Centre. 

EQI  teams  are  promoting  their  projects.  They  are  presenting  their  materials  to 
teachers  in  their  districts,  at  professional  development  days,  teachers'  conventions 
and  conferences.  Project  teams  and  Alberta  Education  staff  will  provide 
appropriate  inservice  activities  to  colleagues.  Shared  knowledge  becomes 
a  motivator  to  action. 

Alberta  Education  will  provide  leadership  in  promoting  and  extending  the  work  of 
the  EQI  initiative  throughout  the  province.  Strategies  to  accomplish  this  include 
CASS  zone  meetings,  teachers'  conventions,  conferences,  and  workshops. 
Regional  consortia  will  be  promoted  to  integrate  the  products  throughout  the 
province. 

EQI  teams  have  also  expressed  interest  in  continuing  to  network  and  meet  on  a 
regular  basis  to  maintain  the  momentum.  Such  meetings  can  become  accessible 
to  other  districts  wishing  to  participate. 
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Several  issues  remain  to  be  resolved.  For  example,  the  EQI  partners  were  unable 
to  agree  on  a  definition  of  a  graduation  rate  since  it  can  be  calculated  in  many 
different  ways.  Other  issues  which  remain  include  frequency  of  measurement, 
sampling  of  students,  time,  and  data  storage.  These  will  be  addressed  in  the 
coming  months. 

Alberta  Education  will  now  work  toward  an  integrated  information  system  that 
includes  a  wider  range  of  educational  indicators  to  better  reflect  the  goals  of 
schooling.  The  interpretation  and  recommended  directions  of  the  local  indicator 
projects,  together  with  other  provincial  initiatives,  provides  a  solid  foundation  for 
the  implementation  of  an  information  system  that  measures  the  success  of  the 
educational  enterprise. 


Concluding  Statement 

The  ownership  and  commitment  that  resulted  from  leaders  developing  local 
indicator  systems,  which  reflect  the  values  of  their  respective  communities, 
have  positive  spin-off  effects  on  the  larger  educational  context  in  Alberta.  The 
benefits  of  this  three-year  collaboration  include  the  discussion  among  participants 
on  the  emerging  set  of  indicators,  the  examination  of  issues,  and  the 
communication  of  results  to  diverse  client  groups.  The  information  generated 
from  the  ten  projects  will  assist  Alberta  Education  to  develop  provincial 
indicators,  trie  interpretation  and  recommended  directions  of  the  local  indicator 
projects,  together  with  other  provincial  initiatives,  provides  a  solid  foundation  for 
the  implementation  of  an  information  system  that  measures  the  success  of  the 
educational  enterprise  in  the  province. 

Educational  indicators  can  provide  better  information  to  help  improve  education. 
They  can  provide  a  more  balanced  picture  of  the  operation  and  outcomes  of 
schooling,  particularly  if  they  focus  on  a  broader  range  of  desired  outcomes.  By 
focusing  on  student  learning,  and  interpreting  and  reflecting  on  the  findings,  better 
decisions  about  schooling  will  result. 
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